Imaging apparatus, image processing device, computer-readable medium having stored thereon an imaging apparatus controlling program, and computer-readable medium having stored thereon an image processing program

ABSTRACT

Provided is an imaging apparatus including an imaging section that generates, from a single scene, captured image data including reference image data, first parallax image data having a first parallax in one direction relative to a subject of the reference image data, and second parallax image data having a second parallax in another direction that is opposite the one direction; an adjustment condition acquiring section that acquires an adjustment condition relating to parallax amount adjustment; and an image processing device that processes the reference image data, the first parallax image data, and the second parallax image data, based on the adjustment condition, to generate third parallax image data having a third parallax that is in the one direction and different from the first parallax and fourth parallax image data having a fourth parallax that is in the other direction and different from the second parallax.

The contents of the following Japanese and PCT patent applications areincorporated herein by reference:

NO. 2012-236192 filed on Oct. 26, 2012,

NO. 2012-236193 filed on Oct. 26, 2012, and

NO. PCT/JP2013/006360 filed on Oct. 28, 2013.

BACKGROUND

1. Technical Field

The present invention relates to an imaging apparatus, an imageprocessing device, a program for controlling an imaging apparatus, and aprogram for controlling an image processing device.

2. Related Art

A stereo imaging apparatus is known in which two imaging optical systemsare used to acquire a stereo image formed by an image for a right eyeand an image for a left eye.

Patent Document 1: Japanese Patent Application Publication No. H8-47001

There are cases where raw stereo image data that is captured throughimaging by a stereo imaging device exhibits an extreme parallax betweenthe left and right images due to the conditions at the time of imaging,the arrangement of subjects in the scene, or the like, and the viewerexperiences a feeling of unnaturalness or eye strain when viewing theimage.

SUMMARY

According to a first aspect of the present invention, provided is animaging apparatus comprising an imaging section that generates, from asingle scene, captured image data including reference image data, firstparallax image data having a first parallax in one direction relative toa subject of the reference image data, and second parallax image datahaving a second parallax in another direction that is opposite the onedirection; an adjustment condition acquiring section that acquires anadjustment condition relating to parallax amount adjustment; and animage processing device that processes the reference image data, thefirst parallax image data, and the second parallax image data, based onthe adjustment condition, to generate third parallax image data having athird parallax that is in the one direction and different from the firstparallax and fourth parallax image data having a fourth parallax that isin the other direction and different from the second parallax.

According to a second aspect of the present invention, provided is animaging apparatus comprising an imaging section that generates, from asingle scene, captured image data including reference image data, firstparallax image data having a first parallax in one direction relative toa subject of the reference image data, and second parallax image datahaving a second parallax in another direction that is opposite the onedirection; an adjustment condition acquiring section that acquires anadjustment condition relating to parallax amount adjustment forgenerating adjusted parallax image data having a parallax that isdifferent from the first parallax and the second parallax; and an outputsection that outputs the adjustment condition in association with thecaptured image data.

According to a third aspect of the present invention, provided is animage processing device comprising an obtaining section that obtainscaptured image data including reference image data, first parallax imagedata having a first parallax in one direction relative to a subject ofthe reference image data, and second parallax image data having a secondparallax in another direction that is opposite the one direction, and anadjustment condition relating to a parallax amount adjustment associatedwith the captured image data; and an image processing device thatprocesses the reference image data, the first parallax image data, andthe second parallax image data, based on the adjustment condition, togenerate third parallax image data having a third parallax that is inthe one direction and different from the first parallax and fourthparallax image data having a fourth parallax that is in the otherdirection and different from the second parallax.

According to a fourth aspect of the present invention, provided is acontrol program for an imaging apparatus that causes a computer togenerate, from a single scene, captured image data including referenceimage data, first parallax image data having a first parallax in onedirection relative to a subject of the reference image data, and secondparallax image data having a second parallax in another direction thatis opposite the one direction; acquire an adjustment condition relatingto parallax amount adjustment; and process the reference image data, thefirst parallax image data, and the second parallax image data, based onthe adjustment condition, to generate third parallax image data having athird parallax that is in the one direction and different from the firstparallax and fourth parallax image data having a fourth parallax that isin the other direction and different from the second parallax.

According to a fifth aspect of the present invention, provided is acontrol program for an imaging apparatus that causes a computer togenerate, from a single scene, captured image data including referenceimage data, first parallax image data having a first parallax in onedirection relative to a subject of the reference image data, and secondparallax image data having a second parallax in another direction thatis opposite the one direction; acquire an adjustment condition relatingto parallax amount adjustment for generating adjusted parallax imagedata having a parallax that is different from the first parallax and thesecond parallax; and output the adjustment condition in association withthe captured image data.

According to a sixth aspect of the present invention, provided is acontrol program for an image processing device that causes a computer toobtain captured image data including reference image data, firstparallax image data having a first parallax in one direction relative toa subject of the reference image data, and second parallax image datahaving a second parallax in another direction that is opposite the onedirection, and an adjustment condition relating to a parallax amountadjustment associated with the captured image data; and process thereference image data, the first parallax image data, and the secondparallax image data, based on the adjustment condition, to generatethird parallax image data having a third parallax that is in the onedirection and different from the first parallax and fourth parallaximage data having a fourth parallax that is in the other direction anddifferent from the second parallax.

According to a seventh aspect of the present invention, provided is animaging apparatus comprising a detecting section that detects a subjectdistribution in a depth direction for a scene; a determining sectionthat determines a change condition relating to a parallax amount, basedon the subject distribution; and a control section that generatescaptured image data including first parallax image data and secondparallax image data having a parallax therebetween, based on the changecondition.

According to an eighth aspect of the present invention, provided is acontrol program for an imaging apparatus causing a computer to detect asubject distribution in a depth direction for a scene; determine achange condition relating to a parallax amount, based on the subjectdistribution; and generate captured image data including first parallaximage data and second parallax image data having a parallaxtherebetween, based on the change condition.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a structure of the digital camera 10 according to thepresent embodiment.

FIG. 2 is a perspective view showing the state of an enlarged portion ofthe image capturing element.

FIG. 3 is used to describe an example of the process for generating theparallax image data and the 2D image data.

FIG. 4-A is used to describe the basics of defocus.

FIG. 4-B is used to describe the basics of defocus.

FIG. 4-C is used to describe the basics of defocus.

FIG. 4-D is used to describe the basics of defocus.

FIG. 5 shows an optical intensity distribution output by the parallaxpixels.

FIG. 6 shows a pixel value distribution for describing the basics of theadjusted parallax amount.

FIG. 7 is used to describe the process for generating color parallaxplane data.

FIG. 8-A is used to describe change in RGB pixel value distributions.

FIG. 8-B is used to describe change in RGB pixel value distributions.

FIG. 8-C is used to describe change in RGB pixel value distributions.

FIG. 9 shows the relationship between the parallax amount and the angleof convergence of the viewer.

FIG. 10-A schematically shows the relationship between the diaphragmvalue, the contrast indicating the image sharpness, the distance of thesubject, and the parallax amount.

FIG. 10-B schematically shows the relationship between the diaphragmvalue, the contrast indicating the image sharpness, the distance of thesubject, and the parallax amount.

FIG. 10-C schematically shows the relationship between the diaphragmvalue, the contrast indicating the image sharpness, the distance of thesubject, and the parallax amount.

FIG. 11 is a back surface view of the digital camera displaying a menuscreen for limiting the parallax amount.

FIG. 12-A shows the basics of the parallax amount adjustment.

FIG. 12-B shows the basics of the parallax amount adjustment.

FIG. 12-C shows the basics of the parallax amount adjustment.

FIG. 13 is a process flow when capturing a moving image.

FIG. 14 is a process flow up to the generation of the parallax colorimage data.

FIG. 15 is used to describe preferable aperture shapes.

FIG. 16 is used to describe the connection between the digital cameraand a TV monitor.

FIG. 17 shows a process flow performed when the digital camera capturesa moving image according to the present modification.

FIG. 18 shows a process flow for moving image playback by the TV monitoraccording to the present modification.

FIG. 19 shows a structure of the digital camera according to the presentembodiment.

FIG. 20 schematically shows the relationship between the contrastindicating the image sharpness and the parallax amount.

FIG. 21-A schematically shows the relationship between the subjectdistribution and the parallax amount.

FIG. 21-B schematically shows the relationship between the subjectdistribution and the parallax amount.

FIG. 21-C schematically shows the relationship between the subjectdistribution and the parallax amount.

FIG. 22-A schematically shows the relationship between the diaphragmvalue and the parallax amount.

FIG. 22-B schematically shows the relationship between the diaphragmvalue and the parallax amount.

FIG. 22-C schematically shows the relationship between the diaphragmvalue and the parallax amount.

FIG. 23 schematically shows the basics of a focus shift.

FIG. 24-A shows the basics of a parallax amount adjustment using astereo adjustment parameter.

FIG. 24-B shows the basics of a parallax amount adjustment using astereo adjustment parameter.

FIG. 24-C shows the basics of a parallax amount adjustment using astereo adjustment parameter.

FIG. 25 is a back side view of the digital camera displaying a menuscreen for limiting the parallax amount range.

FIG. 26-A is used to describe subject designation.

FIG. 26-B is used to describe subject designation.

FIG. 27 shows the process flow during the moving image capturingaccording to the first embodiment example.

FIG. 28 shows the process flow during the moving image capturingaccording to the second embodiment example.

FIG. 29 shows the process flow up to the point of generating theparallax color image data.

FIG. 30 is used to describe desirable aperture shapes.

FIG. 31 is used to describe the connection between the digital cameraand a TV monitor.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, some embodiments of the present invention will bedescribed. The embodiments do not limit the invention according to theclaims, and all the combinations of the features described in theembodiments are not necessarily essential to means provided by aspectsof the invention.

First Embodiment

A digital camera according to an embodiment of the present invention,which is an embodiment of an imaging apparatus, is configured in amanner to be able to generate an image of a single scene having aplurality of viewpoints, through a single occurrence of imaging. Eachimage having a different viewpoint from another image is referred to asa parallax image. The present embodiment describes a particular exampleof generating a right parallax image and a left parallax image accordingto two viewpoints that correspond to a right eye and a left eye. Thedigital camera of the present invention can generate both a parallaximage and a non-parallax image that has no parallax from a centralviewpoint.

FIG. 1 shows a structure of the digital camera 10 according to thepresent embodiment. The digital camera 10 includes an imaging lens 20serving as an imaging optical system, and guides subject light that isincident thereto along an optical axis 21 to the image capturing element100. The imaging lens 20 may be an exchangeable lens that can beattached to and detached from the digital camera 10. The digital camera10 includes an image capturing element 100, a control section 201, anA/D conversion circuit 202, a memory 203, a driving section 204, animage processing device 205, a memory card IF 207, a manipulatingsection 208, a display section 209, and an LCD drive circuit 210.

As shown in the drawing, a direction parallel to the optical axis 21 andpointing toward the image capturing element 100 is defined as thepositive direction on the Z axis, a direction pointing toward the readerfrom the plane of the drawing in a plane orthogonal to the Z axis isdefined as the positive direction on the X axis, and a directionpointing toward the top of the drawing in the plane orthogonal to the Zaxis is defined as the positive direction on the Y axis. In several ofthe following drawings, the coordinate axes of FIG. 1 are used as thereference to display the orientation of each drawing.

The imaging lens 20 is formed from a plurality of optical lenses, andfocuses subject light from a scene at a position near a focal plane. Forease of description, FIG. 1 shows a single virtual lens arranged nearthe pupil to represent the imaging lens 20. Furthermore, a diaphragm 22that limits incident light is arranged near the pupil in a manner to beconcentric around the optical axis 21.

The image capturing element 100 is arranged near the focal plane of theimaging lens 20. The image capturing element 100 is an image sensor suchas a CCD or CMOS sensor, in which a plurality of photoelectricconverting elements are arranged two-dimensionally. The image capturingelement 100 experiences timing control from the driving section 204, toconvert a subject image formed on a light receiving surface into animage signal and to output this image signal to the A/D conversioncircuit 202.

The A/D conversion circuit 202 converts the image signal output by theimage capturing element 100 into a digital image signal and outputs thisdigital image signal to the memory 203. The image processing device 205applies various types of image processing with the memory 203 as a workspace, to generate captured image data. The captured image data includesreference image data that is generated from the output of non-parallaxpixels of the image capturing element 100 and parallax image data thatis generated from the output of parallax pixels of the image capturingelement 100, as described further below. In a case where the processingsection used up to the point of generating the captured image data isthe imaging section, the imaging section includes the image capturingelement 100, the A/D conversion circuit 202, the memory 203, the controlsection 201, and the image processing device 205.

The control section 201 performs overall control of the digital camera10. For example, the control section 201 adjusts the opening of thediaphragm 22 according to a set diaphragm value, and causes the imaginglens 20 to move back and forth in the direction of the optical axisaccording to an AF evaluation value. Furthermore, the control section201 detects the position of the imaging lens 20 and is aware of thefocus lens position and the focal distance of the imaging lens 20. Yetfurther, the control section 201 transmits a timing control signal tothe driving section 204 and manages the sequences up to the point whenthe image signal output from the image capturing element 100 isprocessed into the captured image data by the image processing device205.

The control section 201 includes an adjustment condition acquiringsection 231. The adjustment condition acquiring section 231 acquiresvarious adjustment conditions for determining stereo adjustmentparameters, which are described further below. Although described ingreater detail below, one example includes sequentially acquiring thefocal distance and the diaphragm value used as the imaging conditionswhen the captured image data was generated, as the adjustmentconditions.

The image processing device 205 includes an adjustment value determiningsection 232, a calculating section 233, and a moving image generatingsection 234. The adjustment value determining section 232 determines thevalue of the stereo adjustment parameter from the adjustment conditionacquired by the adjustment condition acquiring section 231. Thecalculating section 233 uses the determined stereo adjustment parameterto generate new parallax image data from the captured image data. Themoving image generating section 234 connects the new parallax image datagenerated by the calculating section to generate a 3D moving image file.

The image processing device 205 also fulfills general image processingfunctions, such as adjusting image data according to other selectedimage formats. The generated captured image data is converted into adisplay signal by the LCD drive circuit 210 and displayed in the displaysection 209. The generated captured image data is also recorded in amemory card 220 provided to the memory card IF 207.

FIG. 2 is a perspective view showing the state of an enlarged portion ofthe image capturing element 100. At least 20 million pixels are arrangedin a matrix formation in the pixel region. In the present embodiment, aset of 64 pixels containing 8×8 adjacent pixels forms one basic grid110. Each basic grid 110 includes Bayer arrangements, in which fourpixels are arranged in a 2×2 formation as a reference unit, and theseBayer arrangements are in a 4 (4 arrangement in the X and Y directions.As shown in the drawing, in each Bayer arrangement, green filters (Gfilter) are arranged on the upper left pixel and lower right pixel, ablue filter (B filter) is arranged on the bottom left pixel, and a redfilter (R filter) is arranged on the upper right filter.

The basic grid 110 includes parallax pixels and non-parallax pixels. Theparallax pixels are pixels that receive partial light, from among theincident light passed through the imaging lens 20, that has been shiftedrelative to the optical axis of the imaging lens 20. The parallax pixelsare provided with an aperture mask having a shifted aperture that isshifted from the center of the pixel, such that only this partial lightpasses therethrough. The aperture mask is stacked on the color filter,for example. In the present embodiment, the aperture mask creates twotypes of pixels, which are a parallax Lt pixel that is set such that thepartial light reaches the left side relative to the center of the pixeland a parallax Rt pixel that is set such that the partial light reachesthe right side relative to the center of the pixel. On the other hand,the non-parallax pixels are not provided with an aperture mask, and arepixels that receive the entirety of the incident light passed throughthe imaging lens 20.

The parallax pixels can adopt a variety of configurations for receivingthe partial light shifted from the optical axis, such as a shiftedphotodiode region or a selectively reflective film in which a lightreceiving region and a reflective region are separated, and are notlimited to using the aperture mask. In other words, the parallax pixelsneed only be configured to enable reception of the partial light shiftedfrom the optical axis from among the incident light passed by theimaging lens 20.

The pixels within the basic grid 110 are represented as P_(IJ). Forexample, the top left pixel is P₁₁ and the top right pixel is P₈₁. Asshown in the drawing, the parallax pixels are arranged in the followingmanner.

-   -   P₁₁: Parallax Lt pixel+G filter (=G(Lt))    -   P₅₁: Parallax Rt pixel+G filter (=G(Rt))    -   P₃₂: Parallax Lt pixel+B filter (=B(Lt))    -   P₆₃: Parallax Rt pixel+R filter (=R(Rt))    -   P₁₅: Parallax Rt pixel+G filter (=G(Rt))    -   P₅₅: Parallax Lt pixel+G filter (=G(Lt))    -   P₇₆: Parallax Rt pixel+B filter (=B(Rt))    -   P₂₇: Parallax Lt pixel+R filter (=R(Lt))

The other pixels are non-parallax pixels, and are each one of anon-parallax pixel+R filter, a non-parallax pixel+G filter, and anon-parallax pixel+B filter.

When considering the overall image capturing element 100, the parallaxpixels are divided into a first group having the G filters, a secondgroup having the R filters, and a third group having the B filters, andeach basic grid 110 includes at least one of a parallax Lt pixel and aparallax Rt pixel associated with each of these groups. As shown in theexample in the drawing, the parallax pixels and non-parallax pixels maybe arranged randomly within the basic grid 110. By arranging the pixelsrandomly, the RGB color information can be acquired as the output of theparallax pixels without causing a skew in the spatial resolution foreach color component, and therefore high-quality parallax image data canbe acquired.

The following describes the basics of the process for generating theparallax image data and the 2D image data from the captured image dataoutput from the image capturing element 100. FIG. 3 is used to describean example of the process for generating the parallax image data and the2D image data.

As understood from the arrangement of parallax pixels and non-parallaxpixels in the basic grid 110, even when the output of the imagecapturing element 100 is compiled in a manner matching the pixelarrangement, this will not result in image data expressing a specifiedimage. The pixel output of the image capturing element 100 is gatheredtogether according to pixel groups obtained by separating the pixelsinto groups with the same characteristic, and then image datarepresenting one image corresponding to this characteristic is formed.For example, when each of the left and right parallax pixels aregathered together, left parallax image data and right parallax imagedata having a parallax therebetween is obtained. In this way, the pixelsare divided into pixel groups each having the same characteristic andgathered together into image data, which is referred to as plane data.

The image processing device 205 receives RAW image data in which theoutput values (pixel values) are compiled in an order corresponding tothe pixel arrangement of the image capturing element 100, and a planeseparation process is performed to separate the RAW image data into aplurality of pieces of plane data. The left column in the drawing showsan example of a process for generating 2D-RGB plane data as the 2D imagedata.

When generating the 2D-RGB plane data, the image processing device 205first deletes the pixel values of the parallax pixels to create emptypixels. The pixel values for these empty pixels are calculated accordingto an interpolation process using pixel values of surrounding pixels.For example, the pixel value of the empty pixel P₁₁ is calculated byaveraging the pixel values of P⁻¹⁻¹, P₂₋₁, P⁻¹², and P₂₂, which are thepixel values of the G filter pixels diagonally adjacent to the emptypixel P₁₁. As another example, the pixel value of the empty pixel P₆₃ iscalculated by averaging the pixel values of P₄₃, P₆₁, P₈₃, and P₆₅,which are the pixel values of the R filter pixels separated by one pixelfrom the empty pixel P₆₃ in the horizontal and vertical directions. Inthe same manner, the pixel value of the empty pixel P₇₆ is calculated byaveraging the pixel values of P₅₆, P₇₄, P₉₆, and P₇₈, which are thepixel values of the B filter pixels separated by one pixel from theempty pixel P₇₆ in the horizontal and vertical directions.

The 2D-RGB plane data interpolated in this way is the same as the outputof a normal image capturing element having a Bayer arrangement, andtherefore various types of processing can be performed with 2D imagedata after this. In other words, the widely known Bayer interpolation isperformed to generate color image data in which RGB data is complete foreach pixel. The image processing device 205 performs image processingfor a general 2D image according to a predetermined format, such as JPEGin a case where still image data is being generated or MPEG in a casewhere moving image data is being generated.

In the present embodiment, the image processing device 205 furtherdivides the 2D-RGB plane data according to color and applies theinterpolation process described above to generate each type of planedata as the reference image data. In other words, the image processingdevice 205 generates three types of plane data including Gn plane dataas the green reference image plane data, Rn plane data as the redreference image plane data, and Bn plane data as the blue referenceimage plane data.

The right column in the drawing shows an example of the process forgenerating two pieces of G plane data, two pieces of R plane data, andtwo pieces of B plane data as the parallax pixel data. The two pieces ofG plane data include GLt plane data as the left parallax image data andGRt plane data as the right parallax image data, the two pieces of Rplane data include RLt plane data as the left parallax image data andRRt plane data as the right parallax image data, and the two pieces of Bplane data include BLt plane data as the left parallax image data andBRt plane data as the right parallax image data.

When generating the GLt plane data, the image processing device 205deletes the pixel values other than the pixel values of the G(Lt) pixelsfrom among all output values of the image capturing element 100, therebycreating empty pixels. Accordingly, the two pixel values of P₁₁ and P₅₅remain in the basic grid 110. The basic grid 110 is divided horizontallyand vertically into four equal portions, with the portion of 16 pixelsin the top left being represented by the output value of P₁₁ and theportion of 16 pixels in the bottom right being represented by the outputvalue of P₅₅. The portion of 16 pixels in the top right and the portionof 16 pixels in the bottom left are interpolated by averaging therepresentative values on the sides adjacent thereto vertically andhorizontally. In other words, the GLt plane data has one value per unitof 16 pixels.

In the same manner, when generating the GRt plane data, the imageprocessing device 205 deletes the pixel values other than the pixelvalues of the G(Rt) pixels from among all output values of the imagecapturing element 100, thereby creating empty pixels. Accordingly, thetwo pixel values of P₅₁ and P₁₅ remain in the basic grid 110. The basicgrid 110 is divided horizontally and vertically into four equalportions, with the portion of 16 pixels in the top right beingrepresented by the output value of P₅₁ and the portion of 16 pixels inthe bottom left being represented by the output value of P₁₅. Theportion of 16 pixels in the top left and the portion of 16 pixels in thebottom right are interpolated by averaging the representative values onthe sides adjacent thereto vertically and horizontally. In other words,the GRt plane data has one value per unit of 16 pixels. In this way, itis possible to generate GLt plane data and GRt plane data having lowerresolution than the 2D-RGB plane data.

When generating the RLt plane data, the image processing device 205deletes the pixel values other than the pixel values of the R(Lt) pixelsfrom among all output values of the image capturing element 100, therebycreating empty pixels. Accordingly, the pixel value of P₂₇ remains inthe basic grid 110. This pixel value is the representative value for the64-pixel portion of the basic grid 110. In the same manner, whengenerating the RRt plane data, the image processing device 205 deletesthe pixel values other than the pixel values of the R(Rt) pixels fromamong all output values of the image capturing element 100, therebycreating empty pixels. Accordingly, the pixel value of P₆₃ remains inthe basic grid 110. This pixel value is the representative value for the64-pixel portion of the basic grid 110. In this way, RLt plane data andRRt plane data having lower resolution than the 2D-RGB plane data isgenerated. In this case, the resolution of the RLt plane data and theRRt plane data is lower than the resolution of the GLt plane data andthe GRt plane data.

When generating the BLt plane data, the image processing device 205deletes the pixel values other than the pixel values of the B(Lt) pixelsfrom among all output values of the image capturing element 100, therebycreating empty pixels. Accordingly, the pixel value of P₃₂ remains inthe basic grid 110. This pixel value is the representative value for the64-pixel portion of the basic grid 110. In the same manner, whengenerating the BRt plane data, the image processing device 205 deletesthe pixel values other than the pixel values of the R(Rt) pixels fromamong all output values of the image capturing element 100, therebycreating empty pixels. Accordingly, the pixel value of P₇₆ remains inthe basic grid 110. This pixel value is the representative value for the64-pixel portion of the basic grid 110. In this way, BLt plane data andBRt plane data having lower resolution than the 2D-RGB plane data isgenerated. In this case, the resolution of the BLt plane data and theBRt plane data is lower than the resolution of the GLt plane data andthe GRt plane data, and is equal to the resolution of the RLt plane dataand the RRt plane data.

In the present embodiment, there are cases where image processing isapplied to the output image data such that the parallax amount betweenthe generated images becomes a target parallax amount. In such a case,the image processing device 205 uses these pieces of plane data togenerate color image data of a left side viewpoint and color image dataof a right side viewpoint. In particular, by adopting the stereoadjustment parameter, the image processing device 205 generates colorimage data in which the parallax amount for a 3D image is adjusted whilemaintaining the blur amount of the 2D color image. Before describing thespecific process, the basic principles of this generation will bedescribed.

FIGS. 4-A to 4-D are used to describe the basics of defocus. Theparallax Lt pixels and the parallax Rt pixels receive subject light thathas passed through one of two parallax virtual pupils set symmetricallyto the optical axis, as partial regions of a lens pupil. In the opticalsystem of the present embodiment, the actual subject light passesthrough the entirety of the lens pupil and therefore, until reaching theparallax pixels, there is no difference between the optical intensitydistributions of the right and left parallax virtual pupils. However,due to the operation of the aperture mask of each parallax pixel, theparallax pixels output image signals obtained by photoelectricallyconverting only the partial light that has passed through the parallaxvirtual pupils. Accordingly, the pixel value distributions indicated bythe output of the parallax pixels may be thought of as having aproportional relationship with respect to the optical intensitydistribution of the partial light that has passed through thecorresponding parallax virtual pupil.

As shown in FIG. 4-A, when an object point that is the subject ispresent at the focal position, the output of each parallax pixelexhibits a steep pixel value distribution centered on the correspondingimaging point pixel, regardless of which of the parallax virtual pupilsthe subject light has passed through. If the parallax Lt pixel isarranged near the imaging point, the output value of the pixelcorresponding to the imaging point is highest and the output values ofthe surrounding pixels drop sharply. Also, if the parallax Rt pixel isarranged near the imaging point, the output value of the pixelcorresponding to the imaging point is highest and the output values ofthe surrounding pixels drop sharply. In other words, whichever of theparallax virtual pupils the subject light passes through, bothdistributions match in that the output value of the pixel correspondingto the imaging point is highest and the output values of the surroundingpixels drop sharply.

On the other hand, as shown in FIG. 4-B, when the object point isshifted from the focal position, in contrast to a case in which theobject point is at the focal position, the peak of the pixel valuedistribution exhibited by the parallax Lt pixels occurs at a positionthat is at a distance in one direction from the pixel corresponding tothe imaging point, and the output value at this peak decreases.Furthermore, the width of pixels having an output value increases. Thepeak of the pixel value distribution exhibited by the parallax Rt pixelsoccurs at a position that is at a distance equal to the shift distanceof the peak of the parallax Lt pixels from the pixel corresponding tothe imaging point, but in a direction opposite the shift directionoccurring with the parallax Lt pixels, and the output value at this peakdecreases in the same manner. Also in the same manner, the width ofpixels having an output value increases. In other words, the same pixeldistributions occur equidistant from the focal position, but have amilder slope compared to a case in which the object point is at thefocal position. As shown in FIG. 4-C, when the object point is shiftedfarther from the focal position, identical distributions occur fartherfrom the focal position and with even milder slopes compared to thestate seen in FIG. 4-B. Essentially, it can be said that as the objectpoint is shifted farther from the focal position, the blur amount andthe parallax amount increase. In other words, the blur amount andparallax amount change in conjunction with each other, according to thedefocus. Specifically, the blur amount and the parallax amount have aone-to-one correspondence.

FIGS. 4-B and 4-C show cases in which the object point is shifted fromthe focal position in a direction to be father away, but in a case wherethe object point is shifted from the focal point in a direction to becloser, as shown in FIG. 4-D, the relative positional relationshipbetween the pixel value distribution exhibited by the parallax Lt pixelsand the pixel value distribution exhibited by the parallax Rt pixelsbecomes inverted relative to the cases shown in FIGS. 4-B and 4-C. Withthis defocus relationship, a viewer who is viewing the parallax imagesperceives a subject that is present behind the focal position as beingfarther away and perceives a subject that is present in front of thefocal position as being closer.

FIG. 5 shows a result obtained by graphing the changes of each pixelvalue distribution described in FIGS. 4-B and 4-C. In the drawing, thehorizontal axis indicates the pixel position, and the center position isthe pixel position corresponding to the imaging point. The vertical axisindicates the output value (pixel value) of each pixel. The output valueis substantially proportional to the optical intensity, as describedabove.

The distribution curve 1804 and the distribution curve 1805 respectivelyindicate the pixel value distribution of the parallax Lt pixels and thepixel value distribution of the parallax Rt pixels in FIG. 4-B. Asunderstood from the drawing, these distributions are linearlysymmetrical with respect to the center position. Furthermore, thecombined distribution curve 1806 formed by adding together thesedistributions exhibits substantially the same shape as the pixel valuedistribution of the non-parallax pixels for the state shown in FIG. 4-B,i.e. the pixel value distribution in a case where all of the subjectlight is received.

The distribution curve 1807 and the distribution curve 1808 respectivelyindicate the pixel value distribution of the parallax Lt pixels and thepixel value distribution of the parallax Rt pixels in FIG. 4-C. Asunderstood from the drawing, these distributions are also linearlysymmetrical with respect to the center position. Furthermore, thecombined distribution curve 1809 formed by adding together thesedistributions exhibits substantially the same shape as the pixel valuedistribution of the non-parallax pixels for the state shown in FIG. 4-C.

In the present embodiment, when performing image processing using thestereo adjustment parameter, a virtual pixel value distribution isactually created using the pixel values of the parallax Lt pixels andthe pixel values of the parallax Rt pixels in a pixel distributionobtained by acquiring the output values of the image capturing element100 and performing the interpolation process to fill in the emptypixels. At this time, the parallax amount expressed as the space betweenthe peaks is adjusted while approximately maintaining the blur amountexpressed as the spread of the pixel value distributions. In otherwords, the image processing device 205 of the present embodimentgenerates an image that has a parallax amount adjusted between a 2Dimage generated from the non-parallax pixels and a 3D image generatedfrom the parallax pixels, while maintaining the blur amount of the 2Dimage in a practically unaltered state. FIG. 6 shows the pixel valuedistributions for describing the basics of the adjusted parallax amount.

The Lt distribution curve 1901 and the Rt distribution curve 1902 shownby solid lines in the drawing are distribution curves obtained byplotting the actual pixel values of the Lt plane data and the Rt planedata. For example, these curves correspond to the distribution curves1804 and 1805 in FIG. 5. Furthermore, the distance between the peaks ofthe Lt distribution curve 1901 and the Rt distribution curve 1902represents the 3D parallax amount, and when this distance is larger thestereoscopic feeling when playing the image is stronger.

The 2D distribution curve 1903 is obtained by adding together 50% of theLt distribution curve 1901 and 50% of the Rt distribution curve 1902,and has a convex shape without skewing to the right or left. The 2Ddistribution curve 1903 corresponds to a shape that has half the heightof the combined distribution curve 1806 in FIG. 5. In other words, theimage based on this distribution is a 2D image with a parallax amount ofzero.

The adjusted Lt distribution curve 1905 is a curve obtained by addingtogether 80% of the Lt distribution curve 1901 and 20% of the Rtdistribution curve 1902. The peak of the adjusted Lt distribution curve1905 is displaced toward the center more than the peak of the Ltdistribution curve 1901, by an amount corresponding to the addition ofthe Rt distribution curve 1902 component. In the same manner, theadjusted Rt distribution curve 1906 is a curve obtained by addingtogether 20% of the Lt distribution curve 1901 and 80% of the Rtdistribution curve 1902. The peak of the adjusted Rt distribution curve1906 is displaced toward the center more than the peak of the Rtdistribution curve 1902, by an amount corresponding to the addition ofthe Lt distribution curve 1901 component.

Accordingly, the adjusted parallax amount expressed by the distancebetween the peaks of the adjusted Lt distribution curve 1905 and theadjusted Rt distribution curve 1906 is smaller than the 3D parallaxamount. Therefore, the stereoscopic feeling when playing the image isweakened. On the other hand, the spread of the distribution of theadjusted Lt distribution curve 1905 and the distribution of the adjustedRt distribution curve 1906 is equal to the spread of the 2D distributioncurve 1903, and therefore it can be said that the blur amount is equalto that of the 2D image.

In other words, it is possible to control the adjusted parallax amountaccording to what percentages of the Lt distribution curve 1901 and theRt distribution curve 1902 are added together. By applying this adjustedpixel value distribution to each plane of the color image data generatedfrom the non-parallax pixels, it is possible to generate color imagedata of a left side viewpoint and color image data of a right sideviewpoint that provide a stereoscopic feeling different from that of theparallax image data generated from the parallax pixels.

In the present embodiment, the color image data of the left sideviewpoint and the color image data of the right side viewpoint aregenerated from the nine pieces of plane data described using FIG. 3. Thecolor image data of the left side viewpoint is formed by three pieces ofcolor parallax plane data including the RLt_(c) plane data that is redplane data corresponding to the left side viewpoint, the GLt_(c) planedata that is green plane data corresponding to the left side viewpoint,and the BLt_(c) plane data that is blue plane data corresponding to theleft side viewpoint. In the same manner, the color image data of theright side viewpoint is formed by three pieces of color parallax planedata including the RRt_(c) plane data that is red plane datacorresponding to the right side viewpoint, the GRt_(c) plane data thatis green plane data corresponding to the right side viewpoint, and theBRt_(c) plane data that is blue plane data corresponding to the rightside viewpoint.

FIG. 7 is used to describe the process for generating color parallaxplane data. In particular, FIG. 7 shows the process for generating theRLt_(c) plane data and the RRt_(c) plane data, which are for the redparallax plane among the color parallax planes.

The red parallax plane is generated using the pixel values of the RLtplane data, the pixel values of the RRt plane data, and the pixel valuesof the Rn plane data described using FIG. 3. As a specific example, whencalculating the pixel value RLt_(mn) of a target pixel position (i_(m),j_(n)) in the RLt_(c) plane data, the calculating section 233 of theimage processing device 205 first extracts the pixel value RLt_(mn) fromthe same pixel position (i_(m), j_(n)) of the Rn plane data. Next, thecalculating section 233 extracts the pixel value RLt_(mn) from the samepixel position (i_(n), k_(n)) of the RLt plane data and extracts thepixel value RRt_(mn) from the same pixel position (i_(m), j_(n)) of theRRt plane data. The calculating section 233 then multiplies the pixelvalue Rn_(mn) by a value obtained by distributing the pixel valuesRLt_(mn) and RRt_(mn) according to the stereo adjustment parameter C,thereby calculating the pixel value RLt_(cmn). Specifically, thecalculation is performed according to Expression 1 shown below. Here,the stereo adjustment parameter is set in a range of 0.5<C<1.

$\begin{matrix}{{Expression}\mspace{14mu} 1} & \; \\{{RLt}_{cmn} = {2{Rn}_{mn} \times \frac{{C \times {RLt}_{mn}} + {\left( {1 - C} \right) \times {RRt}_{mn}}}{{RLt}_{mn} + {RRt}_{mn}}}} & (1)\end{matrix}$

In the same manner, when calculating the pixel value RRt_(cmn) of atarget pixel position (i_(m), j_(n)) in the RRt_(c) plane data, thecalculating section 233 multiplies the extracted pixel value Rn_(mn) bya value obtained by distributing the pixel value RLt_(mn) and the pixelvalue RRt_(mn) according to the stereo adjustment parameter C, therebycalculating the pixel value RRt_(cmn). Specifically, the calculation isperformed according to Expression 2 shown below.

$\begin{matrix}{{Expression}\mspace{14mu} 2} & \; \\{{RRt}_{cmn} = {2{Rn}_{mn} \times \frac{{\left( {1 - C} \right) \times {RLt}_{mn}} + {C \times {RRt}_{mn}}}{{RLt}_{mn} + {RRt}_{mn}}}} & (2)\end{matrix}$

The calculating section 233 sequentially performs this type of processfrom the pixel (1, 1) at the top left edge to the pixel (i₀, j₀) at thebottom right edge.

Upon finishing the process for generating the RLt_(c) plane data and theRRt_(c) plane data, which is the red parallax plane, the process forgenerating the GLt_(c) plane data and the GRt_(c) plane data, which isthe green parallax plane, is then performed. Specifically, instead ofextracting the pixel value Rn_(mn) from the same pixel position (i_(m),j_(n)) of the Rn plane data as described above, the pixel value Gn_(mn)is extracted from the same pixel position (i_(m), j_(n)) of the Gn planedata. Furthermore, instead of extracting the pixel value RLt_(mn) fromthe same pixel position (i_(m), j_(n)) of the RLt plane data, the pixelvalue GLt_(mn) is extracted from the same pixel position (i_(m), j_(n))of the GLt plane data. In the same manner, instead of extracting thepixel value RRt_(mn) from the same pixel position (i_(m), j_(n)) of theRRt plane data, the pixel value GRt_(mn) is extracted from the samepixel position (i_(m), j_(n)) of the GRt plane data. Processing isperformed in the same manner, except that the parameters of Expression 1and Expression 2 are altered as desired.

Furthermore, upon finishing the process for generating the GLt_(c) planedata and the GRt_(c) plane data, which is the green parallax plane, theprocess for generating the BLT_(c) plane data and the BRt_(c) planedata, which is the blue parallax plane, is then performed. Specifically,instead of extracting the pixel value Rn_(mn) from the same pixelposition (i_(m), j_(n)) of the Rn plane data as described above, thepixel value Bn_(mn) is extracted from the same pixel position (i_(m),j_(n)) of the Bn plane data. Furthermore, instead of extracting thepixel value RLt_(mn) from the same pixel position (i_(m), j_(n)) of theRLt plane data, the pixel value BLt_(mn) is extracted from the samepixel position (i_(m), j_(n)) of the BLt plane data. In the same manner,instead of extracting the pixel value RRt_(mn) from the same pixelposition (i_(m), j_(n)) of the RRt plane data, the pixel value BRt_(mn)is extracted from the same pixel position (i_(m), j_(n)) of the BRtplane data. Processing is performed in the same manner, except that theparameters of Expression 1 and Expression 2 are altered as desired.

With the process described above, color image data of the left sideviewpoint (the RLt_(c) plane data, GLt_(c) plane data, and BLt_(c) planedata) and color image data of the right side viewpoint (the RRt_(c)plane data, GRt_(c) plane data, and BRt_(c) plane data) are generated.In other words, color image data of the right side viewpoint and of theleft side viewpoint can be generated with a relatively simple process,as virtual output that does not depend on the actual pixels of the imagecapturing element 100.

Furthermore, since the stereo adjustment parameter can be changed in arange of 0.5<C<1, the magnitude of the parallax amount can be adjustedfor a 3D image, while maintaining the blur amount of the 2D color imageresulting from the non-parallax pixels. Accordingly, when these types ofimage data are played by a playing device adapted for 3D images, theviewer of the stereo image display panel can view a 3D image adjusted tohave a suitable stereoscopic feeling as a color image. In particular,since the processing is simple, the image data can be generated quicklyand can therefore be used for moving images as well.

The following describes the above processing from a perspectiveconsidering color and pixel value distribution. FIGS. 8-A to 8-C areused to describe the change in the RGB pixel value distributions. FIG.8-A is a graph in which are arranged the output values for each of theG(Lt) pixels, the G(Rt) pixels, the R(Lt) pixels, the R(Rt) pixels, theB(Lt) pixels, the B(Rt) pixels, in a case where white subject light isreceived from a position shifted by a prescribed amount from the focalposition.

FIG. 8-B is a graph in which are arranged the output values for each ofthe R(N) pixels, the G(N) pixels, and the B(N) pixels, which arenon-parallax pixels, in a case where white subject light is receivedfrom the object point in FIG. 8-A. This graph can also be said torepresent the pixel value distribution of each color.

When the above process is applied to each corresponding pixel whileC=0.8, the pixel value distributions shown in the graph of FIG. 8-C areobtained. As understood from the drawing, a distribution is obtainedcorresponding to the pixel values of each RGB color.

The following describes the relationship between the viewer and thevideo, in a case where 3D image data is played by a playing apparatus.FIG. 9 shows the relationship between the angle of convergence of theviewer and the parallax amount. The eyes 50 represents the eyes of theviewer, and are represented by a right eye 51 and a left eye 52distanced from each other in the drawing.

Unadjusted image data, for which the parallax amount has not beenadjusted, is played in the display section 40, thereby displaying asubject 61 of a right eye image and a subject 62 of a left eye image.The subject 61 and the subject 62 are the same subject and are presentat the same position shifted from the focal point during imaging, andtherefore the subject 61 and the subject 62 are displayed in the displaysection 40 with a parallax amount D₁ therebetween.

The eyes 50 attempt to view the subject 61 and the subject 62 asmatching images, and therefore the viewer perceives the subject as beingat a position with a floating distance L1 (indicated by a square in thedrawing) where the straight line connecting the right eye 51 to thesubject 61 intersects with the straight line connecting the left eye 52to the subject 62.

The angle of convergence at this time is θ₁, as shown in the drawing. Ingeneral, when the angle of convergence is larger, a sense ofunnaturalness is perceived in the video, and this causes eye strain.Therefore, when performing image processing using the stereo adjustmentparameter according to the present embodiment, adjusted image data isplayed in which the parallax amount has been adjusted according to thestereo adjustment parameter as described above. The drawing shows astate in which the adjusted image data is played overlapping theunadjusted image data.

The subject 71 of the right eye image and the subject 72 of the left eyeimage of the adjusted image data are displayed in the display section40. The subject 71 and the subject 72 are the same subject, and are alsothe same subject as the subjects 61 and 62. The subject 71 and thesubject 72 are displayed in the display section 40 with a parallaxamount D₂ therebetween. The viewer perceives the subject as being at aposition with a floating distance L2 (indicated by a triangle in thedrawing) where the straight line connecting the right eye 51 to thesubject 71 intersects with the straight line connecting the left eye 52to the subject 72.

The angle of convergence at this time is θ₂, which is smaller than θ₁.Accordingly, the viewer does not experience a sense of floating at theedges and the accumulation of eye strain can be reduced. The parallaxamount is suitably adjusted as described further below, and thereforethe viewer can view the video while perceiving a comfortable sense offloating (and a sense of the stereoscopic nature of the video whencombined with a perception of sinking when the defocus relationship isinverted).

The parallax amount used in the description of FIG. 9 is expressed as aseparation distance in the display section 40, but the parallax amountcan be defined in various ways. For example, the parallax amount may bedefined by pixel units in the captured image data or by the width of theshift relative to the horizontal width of the images.

FIGS. 10-A to 10-C schematically show the relationship between thediaphragm value of the digital camera 10, the contrast indicating theimage sharpness, the distance of the subject, and the parallax amount.The horizontal axes represent the distance from the digital camera 10and the vertical axes represent the parallax amount and contrast. FIG.10-A shows a state in which the diaphragm value is F1.4, FIG. 10-B showsa state in which the diaphragm value is F4, and FIG. 10-C shows a statein which the diaphragm value is F8. The focal distance of the imaginglens 20 is the same in each of these states, and the digital camera 10aligns the focal point with the main subject positioned at a distanceL₁₀.

The contrast curve 1610 is highest at the distance L₁₀, which is thedistance to the focal position in each of the states. On the other hand,when the diaphragm value is larger, the contrast is relatively high infront of and behind the focal distance. In other words, an imagecaptured in a state where the diaphragm 22 is more constricted exhibitsgreater depth of field.

The parallax amount curve 1620 indicates a parallax amount of zero atthe distance L₁₀, and has a curve in which the slope is greater whencloser to the digital camera 10 from the distance L₁₀. In other words,the parallax amount curve 1620 exhibits a positive value when closerthan the distance L₁₀, and indicates that closer subjects appear to befloating to a greater degree.

On the other hand, the parallax amount curve 1620 has a curve in whichthe slope is smaller when farther from the digital camera 10 than thedistance L₁₀. In other words, the parallax amount curve 1620 exhibits anegative value when farther than the distance L₁₀, and indicates thatfarther subjects appear to be slowly sinking to a greater degree.

Furthermore, the change in the parallax amount curve 1620 becomes moregradual when the diaphragm value becomes greater. In other words,compared to a case in which the diaphragm value is F1.4, the transitionto F4 and F8 results in the parallax amount in front of the focalposition and the parallax amount behind the focal position both becomingsmaller.

If the viewer does not experience a sense of unnaturalness or eye strainwhen the parallax amount is within a range from −m to +m, then theparallax amount curve 1620 is within this range if the diaphragm valueis F8, and therefore the viewer can comfortably view 3D video no matterwhat distance the subject is at.

On the other hand, when the diaphragm value is F1.4 or F4, +m isexceeded on the near-distance side of the parallax amount curve 1620.Accordingly, if the subject is at a distance closer than +m, the viewerexperiences a sense of unnaturalness and eye strain. Therefore, theimage processing device 205 of the present embodiment generates adjustedimage data in which the parallax amount is adjusted by the stereoadjustment parameter to be between a set minimum value and a set maximumvalue.

First, the limits on the parallax amount will be described. FIG. 11 is aback surface view of the digital camera 10 displaying a menu screen forlimiting the parallax amount.

The parallax amount limitations are set as a minimum value −m and amaximum value +m, as described above. The minimum value and the maximumvalue may have different absolute values. Here, the parallax amount isrepresented in pixel units of the parallax image in the adjustedparallax image data.

The parallax amount at which a viewer experiences a sense ofunnaturalness and eye strain differs for each viewer. Accordingly, thedigital camera 10 is preferably configured such that the settings forthe parallax amount limitations can be changed by the image capturer,who is the user of the digital camera 10, during image capturing.

The digital camera 10 is provided with four selections such as shown inthe drawing, for example, as the parallax amount limitation menu.Specifically, these four options include “standard,” where a range inwhich a viewer can usually view images comfortably is preset, “strong,”where a range that is wider than the standard range is preset to allowfor a greater parallax amount, “weak,” where a range that is narrowerthan the standard range is preset to allow for only a small parallaxamount, and “manual,” where the image capturer inputs numerical valuesfor the minimum value and maximum value. When “manual” is selected, theimage capturer can sequentially set, in pixel units, the “maximumfloating amount” as the maximum value and the “maximum sinking amount”as the minimum value. By manipulating the dial button 2081, which is aportion of the manipulating section 208, the image capturer candesignate one of these selections.

When the allowable parallax amount range is set as the parallax amountlimitation in this manner, the image processing device 205 generates theadjusted parallax image data in which the parallax amount is adjusted tobe within this range. The parallax amount adjustment process of thepresent embodiment does not require a complex process such as seen inthe conventional art, where each subject object is cut out and movedhorizontally in object units while using a depth map. Accordingly,calculation can be performed more quickly than in the conventional art,and therefore the image processing device 205 can be easily adapted tooutput adjusted parallax image data in real time, even for moving imagesin which the state of the subject changes over time.

The digital camera 10 of the present embodiment has, as one type ofmoving image capturing mode, an automatic 3D moving image mode tocontinuously generate parallax image data adjusted to have a comfortableparallax amount and connect these pieces of data to generate a movingimage file. Prior to image capturing, the image capturer selects thisautomatic 3D moving image mode by manipulating the mode button 2082,which is a portion of the manipulating section 208.

Whether all of the subjects forming a single scene fall within the setparallax amount range depends on a variety of conditions, as isunderstood from the descriptions of FIGS. 10-A to 10-C. Theseconditions, including the set parallax amount range as well, are set asadjustment conditions relating to the parallax amount adjustment.

The following further describes the adjustment conditions. Theadjustment condition acquiring section 231 acquires various adjustmentconditions and passes these adjustment conditions to the adjustmentvalue determining section 232 as appropriate. As described above, theadjustment condition acquiring section 231 acquires, as an adjustmentcondition, the parallax amount range input via the menu screen and themanipulating section 208.

FIGS. 10-A to 10-C describe a case in which the focal distance of theimaging lens 20 is fixed, but when capturing images of a single subject,the parallax amount changes according to the focal distance of theimaging lens 20. In other words, the focal distance, which is a settingcondition of the imaging lens 20 serving as the optical system, canbecome an adjustment condition that affects the parallax amount.Accordingly, the adjustment condition acquiring section 231 acquiresfocal distance information, i.e. zoom information, that is acquired bythe control section 201 from the imaging lens 20, as an adjustmentcondition.

As described using FIGS. 10-A to 10-C, the slope of the parallax amountcurve 1620 changes according to the change in the diaphragm value. Inother words, the diaphragm value when acquiring the captured image data,i.e. during image capturing, can become an adjustment condition thataffects the parallax amount, as a setting condition of the opticalsystem. Accordingly, the adjustment condition acquiring section 231acquires, as an adjustment condition, the diaphragm value acquired bythe control section 201 from the imaging lens 20.

As described using FIGS. 10-A to 10-C, the parallax amount is zero for asubject aligned with the focal point, and takes on a positive ornegative value in front of or behind the focal point. In other words,the focus lens position, which is a setting condition of the imaginglens 20 serving as the optical system, can become an adjustmentcondition affecting the parallax amount. Accordingly, the adjustmentcondition acquiring section 231 acquires, as an adjustment condition,the focus lens position, i.e. the focus information, that is acquired bythe control section 201 from the imaging lens 20.

In FIGS. 10-A to 10-C, it is assumed that the main subject is present atthe distance L₁₀, which is the focal position, but in a case where othersubjects are distributed at different depths, these other subjectsappear to float or sink. In other words, the subject distribution in thedepth direction can become an adjustment condition that affects theparallax amount, as a subject state. Accordingly, the adjustmentcondition acquiring section 231 acquires the subject distribution in thedepth direction as an adjustment condition. Specifically, the controlsection 201 uses the defocus information used for the autofocus todetect the subject distribution, from the defocus amount in each of aplurality of divided regions. The defocus information may utilize theoutput of a phase difference sensor provided especially for this purposeor the output of the parallax pixels of the image capturing element 100.When using the output of the parallax pixels, the parallax image dataprocessed by the image processing device 205 can be used.

The following describes the adjustment of the parallax amount using astereo adjustment parameter whose value is determined according to theadjustment conditions. FIGS. 12-A to 12-C show the basics of theparallax amount adjustment.

FIG. 12-A corresponds to a view obtained by removing the contrast curve1610 from FIG. 10-A. Here, in addition to the in-focus subject servingas the main subject, which is the target image aligned with the focalpoint, it is assumed that there are also a close subject that is infront of the focal point, i.e. on the digital camera 10 side, and a farsubject that is behind the focal point, i.e. on the opposite side. InFIG. 10-A the in-focus subject is at the distance L₁₀, the close subjectis at the distance L₂₀, and the far subject is at the distance L₃₀.

When the parallax amount range is set from −m to +m as an adjustmentcondition, the value of the parallax amount curve 1620 at the distanceL₃₀ where the far subject is located is within this parallax amountrange, and therefore the parallax amount need not be adjusted on the farsubject side. However, the value of the parallax amount curve 1620 atthe distance L₂₀ where the close subject is located exceeds +m, andtherefore the overall parallax amount is adjusted such that the parallaxamount of the image of the close subject becomes +m.

In other words, the parallax amount curve is adjusted such that theparallax amount of the image of the subject located farthest forwardfrom the in-focus subject, i.e. the close subject, and the parallaxamount of the image of the subject located farthest backward from thein-focus subject, i.e. the far subject, are both within the set parallaxamount range. More specifically, the parallax amount curve should beadjusted such that the parallax amount of the subject image that isfurthest outside of the set parallax amount range becomes a limit valueof the parallax amount range. In FIG. 12-A, the adjusted parallax amountcurve 1630 is the result of adjusting the parallax amount curve in thismanner. By performing this adjustment, the images of all subjectsincluded in the same scene fall within the set parallax amount range.

FIG. 12-B shows the basics of the parallax amount adjustment in a casewhere the in-focus subject is moved deeper from the distance L₁₀ to thedistance L₁₁, from the subject state shown in FIG. 12-A. In this case,the focal position is at the distance L₁₁, and therefore the parallaxamount for the image of the close subject at the distance L₂₀, which hasnot moved, becomes significantly greater than in FIG. 12-A, as shown bythe parallax amount curve 1620. In this case as well, in the same manneras described in FIG. 12-A, the overall parallax amount is adjusted suchthat the parallax amount of the image of the close subject becomes +m.It should be noted that the adjustment amount is greater than theadjustment amount used in the case of FIG. 12-A. As a result, the slopeof the adjusted parallax amount curve 1630 becomes closer to beinghorizontal, and therefore the parallax amount of the image on the farsubject side is more restricted.

FIG. 12-C shows the basics of the parallax amount adjustment in a casewhere the close subject is moved deeper from the distance L₂₀ to thedistance L₂₁, from the subject state shown in FIG. 12-B. In this case,the focal position remains at the distance L₁₁, and therefore theparallax amount curve 1620 remains the same, but since the close subjecthas been shifted deeper, the adjustment amount is less than theadjustment amount used in the case of FIG. 12-B.

In the manner described above, the adjustment amount is uniquelydetermined if the adjustment conditions described above can be acquired.The adjustment amount has a one-to-one relationship with the stereoadjustment parameter C, and therefore the adjustment value determiningsection 232 can determine the value of the stereo adjustment parameter Cif the adjustment conditions are received from the adjustment conditionacquiring section 231. Specifically, a look-up table corresponding toFIGS. 12-A to 12-C is prepared in advance, and when each value for theadjustment conditions is input and the look-up table is referenced, theadjustment value determining section 232 can extract and determine thevalue of the stereo adjustment parameter C for this input. The look-uptable is constructed using the results of actual experimentation or asimulation performed in advance, for example. As another example,instead of using a look-up table format, a multivariable function inwhich each value of the adjustment conditions serves as a variable maybe prepared in advance.

The adjustment conditions need not include all of the conditionsdescribed above, and may adopt a portion of these conditions. Forexample, by using only the focal distance of the imaging lens 20 as anadjustment condition and setting a one-to-one relationship between thefocal distance and the value of the stereo adjustment parameter C, thesense of unnaturalness and eye strain experienced by the viewer can bedecreased by some degree.

The following describes a series of process flows of the digital camera10. FIG. 13 shows a process flow performed when capturing a movingimage. The process flow in the drawing begins at the time when the modebutton 2082 is manipulated by the image capturer to initiate theautomatic 3D moving image mode. The parallax amount range is set earlierby the image capturer.

When the automatic 3D moving image mode begins, at step S11, theadjustment condition acquiring section 231 acquires the parallax amountrange set by the image capturer from the system memory. If instructionsother than the parallax amount range are received from the imagecapturer as adjustment conditions, these adjustment conditions are alsoacquired.

At step S12, the control section 201 waits for recording startinstructions made by the image capturer pressing down the recordingstart button. When the recording start instructions are detected (YES instep S12), the control section 201 proceeds to step S13 and performs AFand AE. Then, at step S14, the control section 201 performs chargeaccumulation of the image capturing element 100 via the driving section204 and performs reading, to acquire the captured image data of oneframe. During this time, the control section 201 may continue drivingthe diaphragm 22 and driving the focus lens, according to the detectionresults of step S13.

At step S15, the adjustment condition acquiring section 231 acquires theadjustment conditions in conjunction with acquiring the captured imagedata of step S14. Depending on the type of adjustment conditions, thisacquisition may be performed before step S14 or in parallel with stepS14.

The process then proceeds to step S16, where the adjustment conditionacquiring section 231 passes the acquired adjustment conditions to theadjustment value determining section 232 and the adjustment valuedetermining section 232 references the look-up table, with the receivedadjustment conditions as arguments, to determine the stereo adjustmentparameter C.

At step S17, the calculating section 233 receives the captured imagedata and the stereo adjustment parameter C determined by the adjustmentvalue determining section 232, and generates the color image data of theleft side viewpoint (the RLt_(c) plane data, the GLt_(c) plane data, andthe BLt_(c) plane data) and the color image data of the right sideviewpoint (the RRt_(c) plane data, the GRt_(c) plane data, and theBRt_(c) plane data). The details of this process are described furtherbelow.

At step S18, when it is determined that recording stop instructions havenot been received from the image capturer, the control section 201returns to step S13 and performs processing on the next frame. When itis determined that recording stop instructions have been received, theprocess moves to step S19.

At step S19, the moving image generating section 234 connects thecontinuously generated pieces of color image data for the left sideviewpoint and color image data for the right side viewpoint, andperforms a formatting process according to a 3D-compatible moving imageformat such as Blu-ray(registered trademark) 3D to generate the movingimage file. The control section 201 then records the generated movingimage file to the memory card 220 via the memory card IF 207, and thisprocess flow is finished. The recording to the memory card 220 may beperformed sequentially in synchronization with the generation of thecolor image data for the left side viewpoint and color image data forthe right side viewpoint, and an end-of-file process may be performed insynchronization with the recording stop instructions. Furthermore, thecontrol section 201 is not limited to recording on the memory card 220,and may be configured to output data to an external device via a LAN,for example.

The following provides a detailed description of the process performedat step S17 of FIG. 13. FIG. 14 shows the process flow of step S17, upto the point of generating the parallax color image data, which is thecolor image data of the left side viewpoint and the color image data ofthe right side viewpoint.

At step S101, the calculating section 233 acquires the captured imagedata. Then, at step S102, as described using FIG. 3, the captured imagedata is divided into planes of the non-parallax image data and theparallax image data. At step S103, as described using FIG. 3, thecalculating section 233 performs the interpolation process tointerpolate the empty pixels in each piece of plane data resulting fromthe division.

At step S104, the calculating section 233 initializes each of thevariables. Specifically, first, the color variable Cset is set to 1. Thecolor variable Cset is such that 1=red, 2=green, and 3=blue.Furthermore, the coordinate variables i and j are both set to 1. Yetfurther, the parallax variable S is set to 1. The parallax variable S issuch that 1=left and 2=right.

At step S105, the calculating section 233 extracts the pixel value fromthe target pixel position (i, j) of the Cset plane. For example, whenCset=1 and the target pixel position is (1, 1), the extracted pixelvalue is Rn₁₁. Furthermore, at step S106, the calculating section 233extracts the pixel values from the target pixel position (i, j) of theLt_(Cset) plane data and the Rt_(Cset) plane data. For example, when thetarget pixel position is (1, 1), the extracted pixel values areLt_(Cset11) and Rt_(Cset11).

At step S107, the calculating section calculates the pixel value for thetarget pixel position (i, j) corresponding to the parallax variable S.For example, when Cset=1, S=1, and the target pixel position is (1, 1),RLt_(C11) is calculated. As a specific example, the calculation may beperformed according to Expression 1 shown above. Here, the stereoadjustment parameter C is the value determined at step S16.

At step S108, the calculating section 233 increments the parallaxvariable S. Then, at step S109, it is determined whether the parallaxvariable S has exceeded 2. If the parallax variable S has not exceeded2, the process returns to step S107. If the parallax variable S hasexceeded 2, the process moves to step S110.

At step S110, the calculating section 233 sets the parallax variable Sto 1 and increments the coordinate variable i. Then, at step S111, it isdetermined whether the coordinate variable i has exceeded i₀. If thecoordinate variable i has not exceeded i₀, the process returns to stepS105. If the coordinate variable i has exceeded i₀, the process moves tostep S112.

At step S112, the calculating section 233 sets the coordinate variable ito 1 and increments the coordinate variable j. Then, at step S113, it isdetermined whether the coordinate variable j has exceeded j₀. If thecoordinate variable j has not exceeded j₀, the process returns to stepS105. If the coordinate variable j has exceeded j₀, the process moves tostep S114.

Upon reaching step S114, all of the pixel values on the right and leftfor this Cset have been handled, and therefore the calculating section233 arranges these pixel values to generate the plane image data. Forexample, when Cset=1, the RLt_(c) plane data and RRt_(c) plane data aregenerated.

The process moves to step S115, and the calculating section 233 sets thecoordinate variable j to 1 and increments the color variable Cset. Then,at step S116, it is determined whether the color variable Cset exceeds3. If the color variable Cset does not exceed 3, the process returns tostep S105. If the color variable Cset exceeds 3, then all of the colorimage data of the left side viewpoint (the RLt_(c) plane data, theGLt_(c) plane data, and the BLt_(c) plane data) and the color image dataof the right side viewpoint (the RRt_(c) plane data, the GRt_(c) planedata, and the BRt_(c) plane data) has been collected, and the flowreturns to FIG. 13.

The following describes desirable shapes for the apertures of theaperture mask described using FIG. 2. FIG. 15 shows desirable apertureshapes.

The aperture section 105 of the parallax Lt pixel and the aperturesection 106 of the parallax Rt pixel are preferably shifted in oppositedirections from each other and include the centers of the correspondingpixels. Specifically, the aperture section 105 and the aperture section106 preferably each contact a virtual center line 322 that passesthrough the center of the pixel or straddles this center line 322.

In particular, as shown in the drawing, the shape of the aperturesection 105 and the shape of the aperture section 106 are preferablyrespectively identical to a shape obtained by dividing the shape of theaperture section 104 of the non-parallax pixel along the center line322. In other words, the shape of the aperture section 104 is preferablyequivalent to a shape obtained by placing the shape of the aperturesection 105 and the shape of the aperture section 106 adjacent to eachother.

In the above description, the arithmetic expressions used by thecalculating section 233 are Expressions 1 and 2, which use a weightedarithmetic mean, but the arithmetic expressions are not limited to thisand various other expressions can be used. For example, when using aweighted geometric mean, Expressions 3 and 4 shown below and expressedin the same manner as Expressions 1 and 2 can be used.

$\begin{matrix}{{Expression}\mspace{14mu} 3} & \; \\{{RLt}_{cmn} = {{Rn}_{mn} \times \left( \frac{{RLt}_{mn}}{{RRt}_{mn}} \right)^{({C - 0.5})}}} & (3) \\{{Expression}\mspace{14mu} 4} & \; \\{{RRt}_{cmn} = {{Rn}_{mn} \times \left( \frac{{RRt}_{mn}}{{RLt}_{mn}} \right)^{({C - 0.5})}}} & (4)\end{matrix}$

In this case, the maintained blur amount is not the blur amount found inthe output of the non-parallax pixels, but is instead the blur amountfound in the output of the parallax pixels.

Furthermore, as other examples of arithmetic expressions, Expressions 5and 6 shown below and expressed in the same manner as Expressions 1 and2 can be used.

$\begin{matrix}{\mspace{79mu}{{Expression}\mspace{14mu} 5}} & \; \\{{RLt}_{cmn} = {{Rn}_{mn} \times \sqrt[3]{2 \times \frac{{C \times {RLt}_{mn}} + {\left( {1 - C} \right) \times {RRt}_{mn}}}{{RLt}_{mn} + {RRt}_{mn}}} \times \sqrt[3]{2 \times \frac{{C \times {GLt}_{mn}} + {\left( {1 - C} \right) \times {GRt}_{mn}}}{{GLt}_{mn} + {GRt}_{mn}}} \times \sqrt[3]{2 \times \frac{{C \times {BLt}_{mn}} + {\left( {1 - C} \right) \times {BRt}_{mn}}}{{BLt}_{mn} + {BRt}_{mn}}}}} & (5) \\{\mspace{79mu}{{Expression}\mspace{14mu} 6}} & \; \\{{RRt}_{cmn} = {{Rn}_{mn} \times \sqrt[3]{2 \times \frac{{\left( {1 - C} \right) \times {RLt}_{mn}} + {C \times {RRt}_{mn}}}{{RLt}_{mn} + {RRt}_{mn}}} \times \sqrt[3]{2 \times \frac{{\left( {1 - C} \right) \times {GLt}_{mn}} + {C \times {GRt}_{mn}}}{{GLt}_{mn} + {GRt}_{mn}}} \times \sqrt[3]{2 \times \frac{{\left( {1 - C} \right) \times {BLt}_{mn}} + {C \times {BRt}_{mn}}}{{BLt}_{mn} + {BRt}_{mn}}}}} & (6)\end{matrix}$

In this case, the terms of the cube root do not change when calculatingeach of GLt_(cmn), GRt_(cmn), BLt_(cmn), and BRt_(cmn).

Yet further, Expressions 7 and 8 may be adopted.

$\begin{matrix}{\mspace{79mu}{{Expression}\mspace{14mu} 7}} & \; \\{{RLt}_{cmn} = {{Rn}_{mn} \times \left( \frac{{RLt}_{mn}}{{RRt}_{mn}} \right)^{\frac{({C - 0.5})}{3}} \times \left( \frac{{GLt}_{mn}}{{GRt}_{mn}} \right)^{\frac{({C - 0.5})}{3}} \times \left( \frac{{BLt}_{mn}}{{BRt}_{mn}} \right)^{\frac{({C - 0.5})}{3}}}} & (7) \\{\mspace{79mu}{{Expression}\mspace{14mu} 8}} & \; \\{{RLt}_{cmn} = {{Rn}_{\;{mn}} \times \left( \frac{{RRt}_{mn}}{{RLt}_{mn}} \right)^{\frac{({C - 0.5})}{3}} \times \left( \frac{{GRt}_{mn}}{{GLt}_{mn}} \right)^{\frac{({C - 0.5})}{3}} \times \left( \frac{{BRt}_{mn}}{{BLt}_{mn}} \right)^{\frac{({C - 0.5})}{3}}}} & (8)\end{matrix}$

In this case as well, the terms of the cube root do not change whencalculating each of GLt_(cmn), GRt_(cmn), BLt_(cmn), and BRt_(cmn).

The following describes the connection with the display apparatus. FIG.16 is used to describe the connection between the digital camera 10 anda TV monitor 80. The TV monitor 80 is formed by the display section 40made from liquid crystal, a memory card IF 81 that receives the memorycard 220 that is removed from the digital camera 10, and a remotecontrol 82 that is manipulated by a hand of the user, for example. TheTV monitor 80 is adapted to display a 3D image. The display format ofthe 3D image is not particularly limited. For example, the right eyeimage and the left eye image may be displayed at separate times, or aninterlaced format may be used in which the right and left eye images areformed as thin vertical or horizontal stripes. As another example, theright and left eye images may be arranged in a side by side manner onone side and another side of the screen.

The TV monitor 80 decodes the moving image file formatted to include thecolor image data of the left side viewpoint and the color image data ofthe right side viewpoint, and displays the 3D image in the displaysection 40. In this case, the TV monitor 80 fulfills the functions of ageneral display apparatus that displays a standardized moving imagefile. However, the TV monitor 80 can also function as an imageprocessing device that realizes at least a portion of the functions ofthe control section 201 described using FIG. 1 and at least a portion ofthe functions of the image processing device 205. Specifically, theadjustment condition acquiring section 231 described in FIG. 1 and theimage processing device including the adjustment value determiningsection 232, the calculating section 233, and the moving imagegenerating section 234 may be incorporated in the TV monitor 80. Withthis configuration, it is possible to realize functional roles otherthan the functional roles realized by the combination of the digitalcamera 10 of the present embodiment described above and the TV monitor80. The following describes such a modification.

In the present modification, the process for generating the adjustedimage data in which the parallax amount is adjusted according to thestereo adjustment parameter is handled on the TV monitor 80 side insteadof on the digital camera 10 side. Accordingly, the digital camera 10need not include the adjustment value determining section 232 and thecalculating section 233 of the configuration shown in FIG. 1. Instead,the adjustment condition acquiring section 231 passes the acquiredadjustment conditions to the moving image generating section 234, andthe moving image generating section 234 associates the receivedadjustment conditions with the corresponding frame while creating themoving image file from the captured image data generated by the imageprocessing device 205. This association may include recording taginformation in the moving image file or generating an association filein which the adjustment conditions are recorded and recording linkinformation to the associated file in the moving image file.

The following describes a detailed operational process of the digitalcamera 10 according to the present modification. FIG. 17 shows a processflow performed when the digital camera 10 captures a moving imageaccording to the present modification. Processes related to theprocesses shown in the process flow of FIG. 13 are given the same stepnumbers, such that different processes and additional processes aredescribed while redundant descriptions are omitted.

At step S21, the adjustment condition acquiring section 231 passes theacquired adjustment conditions to the moving image generating section234, and the moving image generating section 234 associates theseadjustment conditions with the captured image data generated at stepS14.

At step S19 the moving image file is created by connecting the pieces ofcaptured image data that are continuously generated and are respectivelyassociated with the adjustment conditions. The moving image file may bedata at any one of the stages described using FIG. 3, as long as themoving image file includes the parallax image data of the left and rightviewpoints and the reference image data as the captured image data ofthe continuous frames. In other words, the separation process, theinterpolation process, and the plane data process may be performed atstep S14 as processes of the digital camera 10, or some or all of theseprocesses may be performed by the TV monitor 80 serving as the imageprocessing device. The control section 201 outputs the generated movingimage file to the memory card 220, and the series of processes isfinished.

The following describes the processing operation of the TV monitor 80 inthe present modification. FIG. 18 shows a process flow for moving imageplayback by the TV monitor 80 according to the present modification.Processes related to the processes shown in the process flow of FIG. 13are given the same step numbers, such that different processes andadditional processes are described while redundant descriptions areomitted. It should be noted that the TV monitor 80 includes the controlsection having the adjustment condition acquiring section 231 and theimage processing device having the adjustment value determining section232, the calculating section 233, and the moving image generatingsection 234. The control section corresponds to the control section 201described using FIG. 1, and the image processing device corresponds tothe image processing device 205 described using FIG. 1.

Upon detecting 3D image playback instructions, at step S31, the controlsection decodes the moving image file acquired via the memory card IF 81and acquires each piece of plane data from the captured image data.Next, at step S32, the adjustment conditions associated with each pieceof captured image data are read and acquired. The process for acquiringeach piece of plane data at step S31 and the process for acquiring theadjustment conditions at step S32 may be performed one after the otheror in parallel.

At step S16, the adjustment value determining section 232 determines thevalue of the stereo adjustment parameter C, and at step S17, thecalculating section 233 and the moving image generating section 234generate the left and right plane image data in which the parallaxamount is adjusted.

The process moves to step S32, where the control section displays in thedisplay section 40 the 3D image resulting from the generated left andright plane image data. Then, at step S34, the control sectiondetermines if playback stop instructions have been received from theviewer or if all of the image data to be played has been completed and,if neither of these is the case, returns to step S31 and begins theprocess to play the next frame. On the other hand, if playback stopinstructions have been received from the viewer or if all of the imagedata to be played has been completed, the playback process is finished.

In the modification described above, all of the adjustment conditionsare acquired by the digital camera 10 during the image capturing andassociated with the captured image data, but the modification may beconfigured such that during playback on the TV monitor 80 the viewer caninput a portion of the adjustment conditions. For example, the viewercan input the parallax amount range by manipulating the remote control82. The adjustment condition acquiring section 231 of the TV monitor 80acquires the input parallax amount range as an adjustment condition, andthe adjustment value determining section 232 determines the value of thestereo adjustment parameter C according to this parallax amount range.With this configuration, the TV monitor 80 can display a 3D imageaccording to the preferences of each viewer.

In the present embodiment described above, the adjustment conditions areassociated with each piece of captured image data in frame units, butthe association between the adjustment conditions and the captured imagedata is not limited to this. For example, one adjustment condition canbe shared across a predetermined unit of time or by a unit of aplurality of frames. Furthermore, adjustment conditions man be sharedamong a plurality of pieces of captured image data while using, as atiming for changing the adjustment conditions, a timing at which thesetting conditions of the optical system are changed or a timing atwhich the state of the subject changes, for example.

In the present embodiment described above, as described using FIGS. 12-Ato 12-C, the parallax amount curve 1620 is adjusted such that the closesubject and the far subject fall within the set parallax amount range,but the reference for adjusting the parallax amount is not limited tothis. For example, a cumulative parallax amount may be calculated byintegrating the parallax amounts of each of the pixels in the entireframe. If the cumulative parallax amount is positive, then the overallimage will be projected forward, and therefore when the cumulativeparallax amount is greater than a predetermined reference value, theadjusted parallax image data is generated by subtracting a parallaxamount using the stereo adjustment parameter. If the cumulative parallaxamount is negative, a process similar to that performed for a positivevalue may be performed or, since the negative value indicates arelatively small sense of unnaturalness, the parallax image data may begenerated as-is without using the stereo adjustment parameter.

When performing evaluation based on the cumulative parallax amount inthis manner, the image may be divided into a plurality of smallerregions and evaluation may be performed by calculating the cumulativevalue for each of these regions. By performing evaluation in this way,even if there is a region in which the parallax amount is particularlyhigh, the parallax amount can be reduced.

Furthermore, the amount of change in the cumulative parallax amount canbe used as a reference for evaluation. For example, the cumulativeparallax amount changes suddenly when the captured scene changes, but ifa reference value is provided in advance as an allowable amount ofchange, then the parallax amount can be decreased when the changeexceeds this allowable change amount. In this case, a process may beperformed to gradually increase the parallax amount until reaching alimit of the originally allowed parallax amount range.

In the present embodiment described above it is assumed that a movingimage is being captured, but the configuration for outputting theparallax image data in which the parallax amount is adjusted based onthe acquired adjustment conditions could obviously also be applied tostill image capturing. A still image captured in this way does not causean extreme parallax between the left and right images, and thereforedoes not cause a sense of unnaturalness in the viewer.

In the present embodiment described above, the TV monitor 80 isdescribed as one example of an image processing device, but the imageprocessing device can take on a variety of forms. For example, the imageprocessing device may be a device that includes a display section or isconnected to a display section, such as a PC, mobile telephone, or gamedevice.

Each process flow of the present embodiment described above is performedaccording to a control program for controlling the control section. Thecontrol program is recorded in an internal nonvolatile memory and isexpanded as needed in a work memory to perform each process. As anotherexample, a control program recorded on a server is transmitted to eachapparatus via a network and expanded in a work memory to perform eachprocess. As yet another example, a control program recorded on a serveris executed on the server and each apparatus performs the processesaccording to control signals transmitted thereto via a network.

The present embodiment a configuration in which the image capturingelement 100 includes non-parallax pixels and the captured image dataincludes reference image data. However, a configuration may be used inwhich the image capturing element 100 does not include non-parallaxpixels and thus the captured image data is formed from only the parallaximage data. In this case, the adjusted parallax image data is generatedfrom the parallax image data by subtracting a parallax amount using thestereo adjustment parameter. Furthermore, even when the image capturingelement 100 includes non-parallax pixels, the adjusted parallax imagedata may be generated using only the parallax image data.

More specifically, the subject image of the right parallax image datahas a first parallax in a first direction relative to a virtual subjectimage serving as a reference and the subject image of the left parallaximage data has a second parallax in another direction that is oppositethe first direction relative to the virtual subject image. In otherwords, the subject image of the right parallax image data and thesubject image of the left parallax image data have a parallaxtherebetween equal to the sum of the first parallax and the secondparallax. At this time, the image processing device generates, as theadjusted parallax image data, the right parallax image data adjusted tohave a third parallax differing from the first parallax in one directionand the left parallax image data adjusted to have a fourth parallaxdiffering from the second parallax in another direction, using theadjustment conditions.

Second Embodiment

A digital camera according to an embodiment of the present invention,which is an embodiment of an imaging apparatus, is configured in amanner to be able to generate an image of a single scene having aplurality of viewpoints, through a single occurrence of imaging. Eachimage having a different viewpoint from the other image is referred toas a parallax image. The present embodiment describes a particularexample of generating a right parallax image and a left parallax imageaccording to two viewpoints that correspond to a right eye and a lefteye. The digital camera of the present invention can generate both aparallax image and a non-parallax image that has no parallax from acentral viewpoint.

FIG. 19 shows a structure of the digital camera 10 according to thepresent embodiment. The digital camera 10 includes an imaging lens 20serving as an imaging optical system, and guides subject light that isincident thereto along an optical axis 21 to the image capturing element100. The imaging lens 20 may be an exchangeable lens that can beattached to and detached from the digital camera 10. The digital camera10 includes an image capturing element 100, a control section 201, anA/D conversion circuit 202, a memory 203, a driving section 204, animage processing device 205, a memory card IF 207, a manipulatingsection 208, a display section 209, and an LCD drive circuit 210.

As shown in the drawing, a direction parallel to the optical axis 21 andpointing toward the image capturing element 100 is defined as thepositive direction on the Z axis, a direction pointing toward the readerfrom the plane of the drawing in a plane orthogonal to the Z axis isdefined as the positive direction on the X axis, and a directionpointing toward the top of the drawing in the plane orthogonal to the Zaxis is defined as the positive direction on the Y axis. In several ofthe following drawings, the coordinate axes of FIG. 19 are used as thereference to display the orientation of each drawing.

The imaging lens 20 is formed from a plurality of optical lenses, andfocuses subject light from a scene at a position near a focal plane. Forease of description, FIG. 19 shows a single virtual lens arranged nearthe pupil to represent the imaging lens 20. Furthermore, a diaphragm 22that limits incident light is arranged near the pupil in a manner to beconcentric around the optical axis 21.

The image capturing element 100 is arranged near the focal plane of theimaging lens 20. The image capturing element 100 is an image sensor suchas a CCD or CMOS sensor, in which a plurality of photoelectricconverting elements are arranged two-dimensionally. The image capturingelement 100 experiences timing control from the driving section 204, toconvert a subject image formed on a light receiving surface into animage signal and to output this image signal to the A/D conversioncircuit 202.

The A/D conversion circuit 202 converts the image signal output by theimage capturing element 100 into a digital image signal and outputs thisdigital image signal to the memory 203. The image processing device 205applies various types of image processing with the memory 203 as a workspace, to generate captured image data. The captured image data includesreference image data that is generated from the output of non-parallaxpixels of the image capturing element 100 and parallax image data thatis generated from the output of parallax pixels of the image capturingelement 100, as described further below.

The control section 201 performs overall control of the digital camera10. For example, the control section 201 adjusts the opening of thediaphragm 22 according to a set diaphragm value, and causes the imaginglens 20 to move back and forth in the direction of the optical axisaccording to an AF evaluation value. Furthermore, the control section201 detects the position of the imaging lens 20 and is aware of thefocus lens position and the focal distance of the imaging lens 20. Yetfurther, the control section 201 transmits a timing control signal tothe driving section 204 and manages the imaging control up to the pointwhen the image signal output from the image capturing element 100 isprocessed into the captured image data by the image processing device205.

The control section 201 includes a depth information detecting section235 and a determining section 236. The depth information detectingsection 235 detects distribution of subjects in a depth direction in ascene. Specifically the control section 201 uses defocus informationthat is used for the autofocus to detect the subject distribution fromthe defocus amount in each of a plurality of divided regions. Thedefocus information may utilize the output of a phase difference sensorprovided especially for this purpose or the output of the parallaxpixels of the image capturing element 100. When using the output of theparallax pixels, the parallax image data processed by the imageprocessing device 205 can be used. As another example, instead of usingthe defocus information, the subject distribution can be detected bymoving the focus lens back and forth and calculating the AF evaluationvalue based on the contrast AF method in each of the divided regions.

The determining section 236 determines a change condition relating tothe parallax amount, based on the subject distribution detected by thedepth information detecting section 235. A more detailed description isprovided further below, but the determining section 236 determines thediaphragm value as an imaging condition, for example, such that theparallax amount between the output parallax images becomes a targetparallax amount. In this case, the change condition relating to theparallax amount is a diaphragm value indicating the diaphragm 22 and howopen it is.

As described above, the image processing device 205 generates thecaptured image data by processing the image signal output from the imagecapturing element 100. Furthermore, the image processing device 205includes the calculating section 233 and the moving image generatingsection 234. When adjusting the parallax amount using the stereoadjustment parameter, which is described in detail further below, thecalculating section 233 generates new parallax image data through imageprocessing. The moving image generating section 234 connects the piecesof parallax image data to generate a 3D image file.

The image processing device 205 also fulfills general image processingfunctions, such as adjusting image data according to other selectedimage formats. The generated captured image data is converted into adisplay signal by the LCD drive circuit 210 and displayed in the displaysection 209. The generated captured image data is also recorded in thememory card 220 provided to the memory card IF 207.

The manipulating section 208 functions as a portion of a receivingsection that is manipulated by a user to transfer instructions to thecontrol section 201. The manipulating section 208 includes a pluralityof manipulating sections, such as a shutter button that receivesinstructions to start image capturing.

The description of the second embodiment shares many points with thefirst embodiment. Specifically, the description of the second embodimentis the same as the description of the first embodiment from FIGS. 2 to9. Accordingly, this redundant description is omitted, and onlydiffering portions are described.

FIG. 20 schematically shows the relationship between the contrastindicating the image sharpness and the parallax amount. The horizontalaxis represents the distance from the digital camera 10 and the verticalaxis represents the parallax amount and contrast height. The digitalcamera 10 aligns the focal point with the main subject positioned at adistance L_(p).

The contrast curve 1610 is a convex curve that is at the highest at thedistance L_(p), which is the distance to the focal position. In otherwords, the image gradually blurs when moving away from the distanceL_(p) either forward or backward.

The parallax amount curve 1620 indicates a parallax amount of zero atthe distance L_(p), and has a curve in which the slope is greater whencloser to the digital camera 10 from the distance L_(p). In other words,the parallax amount curve 1620 exhibits a positive value when closerthan the distance L_(p), and indicates that closer subjects appear to befloating to a greater degree.

On the other hand, the parallax amount curve 1620 has a curve in whichthe slope is smaller when farther from the digital camera 10 than thedistance L_(p). In other words, the parallax amount curve 1620 exhibitsa negative value when farther than the distance L_(p), and indicatesthat farther subjects appear to be slowly sinking to a greater degree.

If the viewer does not experience a sense of unnaturalness or eye strainwhen the parallax amount is within a range from −m to +m, then thesubjects forming the scene may be distributed between the distance L_(f)(the parallax amount at this time being +m) and the distance L_(r) (theparallax amount at this time being −m). In other words, if the subjectclosest to the digital camera 10 is located at the distance L_(f) andthe subject farthest from the digital camera 10 is located at thedistance L_(r), the viewer can comfortably view a 3D video withoutperforming an adjustment of the parallax amount during the later imageprocessing. On the other hand, if the close subject is located at adistance L_(f), (the parallax amount at this time being +m′) that isfarther forward than the distance L_(f), then the allowable parallaxamount is exceeded and therefore the viewer experiences a sense ofunnaturalness and eye strain.

The following further describes the relationship between the subjectdistribution and the parallax amount. FIGS. 21-A to 21-C schematicallyshow the relationship between the subject distribution and the parallaxamount.

FIGS. 21-A to 21-C correspond to drawings obtained by removing thecontrast curve 1610 from FIG. 20. Here, in addition to the in-focussubject serving as the main subject, which is the target image alignedwith the focal point, it is assumed that there are also a close subjectand a far subject. In FIG. 21-A the in-focus subject is at the distanceL₁₀, the close subject is at the distance L₂₀, and the far subject is atthe distance L₃₀.

When the parallax amount range is set from −m to +m as an adjustmentcondition, the value of the parallax amount curve 1620 at the distanceL₃₀ where the far subject is located is within this parallax amountrange. However, the value of the parallax amount curve 1620 at thedistance L₂₀ where the close subject is located exceeds +m.

FIG. 21-B shows the basics of the parallax amount adjustment in a casewhere the in-focus subject is moved deeper from the distance L₁₀ to thedistance L₁₁, from the subject state shown in FIG. 21-A. In this case,the focal position is at the distance L₁₁, and therefore the parallaxamount for the image of the close subject at the distance L₂₀, which hasnot moved, becomes greater than in FIG. 21-A, as shown by the parallaxamount curve 1620. In other words, the amount extending beyond theallowable range is increased.

FIG. 21-C shows the basics of the parallax amount adjustment in a casewhere the close subject is moved deeper from the distance L₂₀ to thedistance L₂₁, and then farther to the distance L₂₂, from the subjectstate shown in FIG. 21-B. The focal position remains at the distanceL₁₁, and therefore the parallax amount curve 1620 remains the same as inFIG. 21-B, but since the close subject has been shifted deeper, theparallax amount at the time when the close subject is at the distanceL₂₁ extends beyond the allowable range, but the amount of this extensionis less than the extension amount in FIG. 21-B. When the close subjectmoves farther to the distance L₂₂, the parallax amount falls within theallowable range.

In other words, the subject distribution in the depth direction for ascene and the position of the subject aligned with the focal point canbe said to be parameters for determining whether the parallax amount iswithin the set allowable range.

The following describes the relationship between the diaphragm value andthe parallax amount. FIGS. 22A to 22C schematically show therelationship between the diaphragm value and the parallax amount. In thesame manner as in FIG. 20, the horizontal axes represent the distancefrom the digital camera 10 and the vertical axes represent the parallaxamount and contrast height. FIG. 22-A shows a state in which thediaphragm value is F1.4, FIG. 22-B shows a state in which the diaphragmvalue is F4, and FIG. 22-C shows a state in which the diaphragm value isF8. The focal distance of the imaging lens 20 is the same in each ofthese states, and the digital camera 10 aligns the focal point with themain subject positioned at a distance L₁₀.

The contrast curve 1610 is highest at the distance L₁₀, which is thedistance to the focal position in each of the states. On the other hand,when the diaphragm 22 is further constricted, i.e. when the diaphragmvalue is greater, the contrast curve 1610 exhibits a relatively highvalue in front of and behind the focal distance. In other words, animage captured when the diaphragm 22 is in a constricted state has agreater depth of field. The parallax amount curve 1620 indicates aparallax amount of zero at the distance L₁₀, and has a curve in whichthe slope is greater when closer to the digital camera 10 from thedistance L₁₀. On the other hand, the parallax amount curve 1620 has acurve in which the slope is smaller when farther from the digital camera10 than the distance L₁₀.

Furthermore, the change in the parallax amount curve 1620 becomes moregradual when the diaphragm value becomes greater. In other words,compared to a case in which the diaphragm value is F1.4, the transitionto F4 and F8 results in the parallax amount in front of the focalposition and the parallax amount behind the focal position both becomingsmaller. If the viewer does not experience a sense of unnaturalness oreye strain when the parallax amount is within a range from −m to +m,then the parallax amount curve 1620 is within this range if thediaphragm value is F8, and therefore the viewer can comfortably view 3Dvideo no matter what distance the subject is at.

On the other hand, when the diaphragm value is F1.4 or F4, the parallaxamount exceeds +m on the near-distance side of the parallax amount curve1620. Specifically, in the case of F1.4, +m is exceeded in the regioncloser than the distance L₂₄, and in the case of F4, +m is exceeded inthe region closer than the distance L₂₅. In this case, the slope of theparallax amount curve 1620 for F4 is less steep than the slope of theparallax amount curve 1620 for F1.4 and therefore the relationship ofL₂₅<L₂₄ is established. With these diaphragm values, if a subject iscloser than the distance L₂₄ or the distance L₂₅, then the viewer willexperience a sense of unnaturalness and eye strain when viewing thecaptured 3D video.

Therefore, in the present embodiment, the imaging conditions that affectthe parallax amount are changed or the stereo adjustment parameters usedin the image processing are changed such that the parallax amountbetween the generated images becomes a target parallax amount that iswithin an allowable parallax amount range of ±m.

First, the changing of the imaging conditions will be described. Asdescribed using FIGS. 22-A to 22-C the diaphragm value affects theparallax amount, and therefore the diaphragm value may be changedaccording to the detected subject distribution, such that the parallaxamount between output parallax images falls within an allowable parallaxamount range. For example, in the state shown in FIG. 22-A where theinitial diaphragm value is F1.4 and the in-focus subject is at thedistance L₁₀, when the close subject is at the distance L₂₅, theparallax amount thereof exceeds +m. The determining section 236 changesthe diaphragm value from F1.4 to F4, which is a diaphragm value at whichthe parallax amount for the subject at the distance L₂₅ is +m.

Changing of the diaphragm value to a larger value is not limited to acase in which the close subject extends beyond the allowable parallaxamount range, and may also be performed in a case where the far subjectextends beyond the allowable parallax amount range. In a case where theparallax amounts of both the close subject and the far subject have someleeway with respect to the allowable parallax amount, the diaphragmvalue may be changed to be smaller, i.e. the diaphragm 22 may be openedfurther. In this case, the shutter speed can be made faster and the ISOsensitivity can be lowered.

The relationship between the parallax amount curve 1620 and the in-focussubject distance for each diaphragm value is prepared in advance in alook-up table. If the subject distribution and allowable parallax amountare used as input values and the look-up table is referenced, thedetermining section 236 can extract and determine the diaphragm value tobe changed to.

In addition to changing the diaphragm value, the changing of the imagingconditions can include a focus shift technique for changing the focusposition. FIG. 23 schematically shows the basics of a focus shift. Thehorizontal and vertical axes are the same as in FIG. 20.

The contrast curve 1610 and the parallax amount curve 1620 represent acontrast curve and a parallax amount curve occurring when the in-focussubject is at the distance L₁₀ and the focus lens is moved to align thefocal point with the subject. In this case, the peak value of thecontrast curve 1610 exceeds a focus threshold value E_(s) for evaluatingthe focus.

When the close subject is positioned at the distance L₂₇, the parallaxamount thereof is +m₀ according to the parallax amount curve 1620, whichexceeds the allowable parallax amount of +m. Therefore, with the focusshift, the focus lens position is corrected in a range beyond the focusthreshold value E_(s), such that the parallax amount at the distance L₂₇falls within the allowable range.

In the example shown in the drawing, the parallax amount curve 1621 isselected that causes the parallax amount for the close subject to be +m,and the distance L_(p) at which the parallax amount is zero in thisparallax amount curve 1621 is extracted. The focus lens position is thenchanged such that the focal position is at the distance L_(p). Thecontrast curve 1611 is the contrast curve occurring at this time. Sincethe subject is actually at the distance L₁₀, the contrast value for thissubject decreases by Δe, as shown in the drawing. The contrast value atthis time need only exceed the focus threshold value E_(s). The imagecaptured after changing the focus lens position in this manner has aslightly decreased contrast value for the main subject, but the imagethereof can still be evaluated as being in focus and the parallax amountfor the close subject is within the allowable range.

If the contrast value for the distance L_(p) does not exceed the focusthreshold value E_(s), then the focus lens position correction is notallowed. In other words, if the parallax amount for the close subject inthe parallax amount curve 1620 significantly exceeds the allowableparallax amount, then this parallax amount cannot be made to be withinthe allowable range even when the focus lens position is changed withina range exceeding the focus threshold value E_(s). In such a case,another technique may also be used in conjunction with the abovetechnique, such as changing the diaphragm value to be a larger value.

When performing the parallax amount adjustment using the focus shift aswell, a look-up table may be prepared in advance and used to indicatethe relationship between the parallax amount curve and the in-focussubject distance for each diaphragm value. If the subject distributionand the allowable parallax amount are used as input values and thelook-up table is referenced, the determining section 236 can extract anddetermine the distance L_(p). The control section 201 changes the focuslens position according to the distance L_(p). The control section 201determines whether the contrast value obtained as the result of thischange exceeds the focus threshold value E_(s). If the contrast value isdetermined to exceed the focus threshold value E_(s), the imagingsequence continues without alteration. If the contrast value isdetermined to not exceed the focus threshold value E_(s), the focus lensis returned to the original position and the process transitions toanother type of control, such as employing another technique inconjunction with the focus shift. As another option, the determiningsection 236 may calculate the amount of the decrease in the contrastwhen the focal position is moved from L₁₀ to L_(p) without having thecontrol section 201 actually move the focus lens, and the controlsection 201 may determine whether the resulting contrast exceeds thefocus threshold value E_(s). In this case, if a contrast AF method isused, for example, the actual evaluation value acquired previously whenthe focus adjustment was made for the distance L₁₀ can be referenced.

The following describes the change of a stereo adjustment parameter.FIGS. 24-A to 24-C show the basics of a parallax amount adjustment usinga stereo adjustment parameter. FIGS. 24-A to 24-C correspondrespectively to FIGS. 21-A to 21-C.

When the parallax amount range set in FIG. 24-A is from −m to +m, thevalue of the parallax amount curve 1620 at the distance L₃₀ where thefar subject is located is within this parallax amount range, andtherefore there is no need to adjust the parallax amount relating to thefar subject side. However, the value of the parallax amount curve 1620at the distance L₂₀ where the close subject is located exceeds theparallax amount +m, and therefore the overall parallax amount isadjusted.

In other words, the parallax amount curve is adjusted such that theparallax amount of the image of the close subject and the parallaxamount of the image of the far subject are both within the set parallaxamount range. More specifically, the parallax amount curve should beadjusted such that the parallax amount of the subject image that extendsfurthest beyond the set parallax amount range becomes a limit value ofthis parallax amount range. In FIG. 24-A, the adjusted parallax amountcurve 1630 is the result of the parallax amount curve adjusted in thismanner. With this type of adjustment, the images of all subjects formingthe same scene are within the set parallax amount range.

In FIG. 24-B as well, in the same manner as in FIG. 24-A, the overallparallax amount is adjusted such that the image of the close subject hasa parallax amount of +m. However, the adjustment amount is greater thanthe adjustment amount used in FIG. 24-A. As a result, the slope of theadjusted parallax amount curve 1630 becomes closer to horizontal, andtherefore the parallax amount on the far subject side is furtherrestricted.

In the case of FIG. 24-C, the focal position remains at the distanceL₁₁, and therefore the parallax amount curve 1620 remains the same as inFIG. 24-B, but since the close subject has been shifted from thedistance L₂₀ to the distance L₂₁, the adjustment amount is less than theadjustment amount in FIG. 24-B. When the close subject moves fartherfrom L₂₁ to L₂₂, the parallax amount falls within the set parallaxamount range without performing an adjustment.

In the manner described above, the adjustment amount for the parallaxamount curve 1620 is uniquely determined if the subject distribution inthe depth direction within a scene and the position of a subject beingfocused on can be acquired. The adjustment amount has a one-to-onerelationship with the stereo adjustment parameter C, and therefore thedetermining section 236 can determine the value of the stereo adjustmentparameter C if the adjustment conditions are received from the depthinformation detecting section 235 or the like. Specifically, a look-uptable corresponding to FIGS. 24-A to 24-C is prepared in advance, andwhen each value for the adjustment conditions is set as an input valueand the look-up table is referenced, the determining section 236 canextract and determine the value of the stereo adjustment parameter C forthis input. The look-up table is constructed using the results of actualexperimentation or a simulation performed in advance, for example. Asanother example, instead of using a look-up table format, amultivariable function in which each value of the adjustment conditionsserves as a variable may be prepared in advance.

The following describes setting the parallax amount range as theallowable parallax amount range. FIG. 25 is a back side view of thedigital camera displaying a menu screen for limiting the parallax amountrange.

The parallax amount limitations are set as a minimum value −m and amaximum value +m, as described above. The minimum value and the maximumvalue may have different absolute values. Here, the parallax amount isrepresented in pixel units of the parallax image in the adjustedparallax image data.

The parallax amount at which a viewer experiences a sense ofunnaturalness and eye strain differs for each viewer. Accordingly, thedigital camera 10 is preferably configured such that the settings forthe parallax amount limitations can be changed by the image capturer,who is the user of the digital camera 10, during image capturing.

The digital camera 10 is provided with four selections such as shown inthe drawing, for example, as the parallax amount range limitation menu.Specifically, these four options include “standard,” where a range inwhich a viewer can usually view images comfortably is preset, “strong,”where a range that is wider than the standard range is preset to allowfor a greater parallax amount, “weak,” where a range that is narrowerthan the standard range is preset to allow for only a small parallaxamount, and “manual,” where the image capturer inputs numerical valuesfor the minimum value and maximum value. When “manual” is selected, theimage capturer can sequentially set, in pixel units, the “maximumfloating amount” as the maximum value and the “maximum sinking amount”as the minimum value. By manipulating the dial button 2081, which is aportion of the manipulating section 208, the image capturer candesignate one of these selections.

The digital camera 10 of the present embodiment has, as one type ofmoving image capturing mode, an automatic 3D moving image mode tocontinuously generate parallax image data adjusted to have a comfortableparallax amount and connect these pieces of data to generate a movingimage file. Prior to image capturing, the image capturer selects thisautomatic 3D moving image mode by manipulating the mode button 2082,which is a portion of the manipulating section 208.

FIGS. 26-A and 26-B are used to describe subject designation. Inparticular, FIG. 26-A shows a subject distribution in the depthdirection from the digital camera 10 in a certain scene and FIG. 26-B isa back side view of the digital camera 10 displaying a live view of thisscene.

As shown in FIG. 26-A, the scene is formed by a tree 300 (distanceL_(o)), a girl 301 (distance L_(f)), a boy 302 (distance L_(p)), and awoman 303 (distance L_(T)), in the stated order beginning with theclosest to the digital camera 10.

As shown in FIG. 26-B, the live view image of this scene is displayed inthe display section 209. Here, the boy 302 is the in-focus subject. TheAF frame 310, which indicates that the boy 302 is in a focused state, isdisplayed overlapping the image of the boy 302.

In the description up to this point, the closest subject to the digitalcamera 10 has been the close subject. However, the subjects forming ascene are sometimes distributed continuously in the depth direction.Furthermore, there are also subjects that are relatively small in thescene and subjects that are not very important in the scene.Accordingly, it is not necessary for the closest subject in the scene tobe treated as the close subject. For example, in the scene of FIG. 26-A,the three people are the major subjects and it is assumed that theviewer will pay attention to these three people during viewing.Accordingly, the subject images that are to undergo a parallax amountadjustment are the images of these three people, and the other subjectimages may be ignored. Therefore, the control section 201 receivesinstructions from the image capturer concerning which subjects are toundergo the parallax amount adjustment.

The display section 209 displays a title 320, e.g. “please select theclose subject,” indicating a state for receiving user instructions. Inthis state, the user touches the subject image that is to be the closesubject, e.g. the girl 301 in the drawing. The display section 209 isprovided with an overlapping touch panel 2083 as a portion of themanipulating section 208, and the control section 201 acquires theoutput of the touch panel 2083 and determines which of the subjects isthe close subject. In this case, a subject (the tree 300 in the drawing)that is in front of the designated subject is excluded from thedetection targets of the subject distribution. These instructions arenot limited to the close subject, and the same type of instructions maybe received for the far subject.

The depth information detecting section 235 sets the subjects designatedby these user instructions (the subjects from the close subject to thefar subject) as detection targets of the subject distribution. In a caseof capturing a moving image or performing continuous image capturing,when continuously generating the captured image data, the subjectsdesignated at the start of the image capturing are tracked by subjecttracking, and the distances L_(f), L_(p), and L_(r) changing over timemay be acquired.

The following describes the process flow for the moving image capturingof the digital camera 10. As described above, in order to set theparallax amount between the generated images to be within an allowableparallax amount range, there are cases where the imaging conditionsaffecting the parallax amount are changed and cases where the stereoadjustment parameters used in the image processing are changed. Both ofthese methods can be combined, but in the following embodiment examples,these processes are described separately.

First Embodiment Example

As a first embodiment example, a process flow for moving image capturingthat involves changing the imaging conditions affecting the parallaxamount is described. FIG. 27 shows the process flow during the movingimage capturing according to the first embodiment example. The flow inthis drawing begins at the point in time when image capturer manipulatesthe mode button 2082 to initiate the automatic 3D moving image mode. Theparallax amount range is set in advance by the image capturer.

When the automatic 3D moving image mode begins, at step S11, thedetermining section 236 acquires the parallax amount range set by theimage capturer from the system memory. At step S12, the control section201 performs AF and AE. The process then proceeds to step S13 where, asdescribed using FIGS. 26-A and 26-B, for example, the control section201 receives from the user the designation of target subjects via thetouch panel 2083. The process proceeds to step S14, where the depthinformation detecting section 235 detects the subject distribution inthe depth direction from the phase difference information detectedduring the AF operation of step S12, for example, for the subjectsdesignated at step S13.

At step S15, the control section 201 waits for recording startinstructions made by the image capturer pressing down the recordingstart button. When the recording start instructions are detected (YES instep S15), the control section 201 proceeds to step 16. If instructionsare not detected the process returns to step S12. After returning tostep S12, tracking of the designated subjects is performed and theprocesses of steps S13 and S14 may be skipped.

At step S16, the determining section 236 changes the imaging conditions.Specifically, the diaphragm value is changed as described using FIGS.22-A to 22-C or the focus lens position is changed as described usingFIG. 23. As another example, the imaging conditions affecting theparallax amount may be changed such that the parallax amount is withinthe set parallax amount range. For example, if the imaging lens 20 is azoom lens, the focal distance can be changed.

The process moves to step S17, where the control section 201 againperforms AF and AE according to the changed imaging conditions. At stepS18, the control section 201 performs charge accumulation of the imagecapturing element 100 via the driving section 204 and performs reading,to acquire the captured image data of one frame. The parallax amountbetween the parallax images in the captured image data acquired herefall within the set parallax amount range.

At step S19, in consideration of a case where the image capturer desiresto change the target subjects during the moving image capturing, thecontrol section 201 receives from the user a designation of the targetsubjects. The process moves to step S20, where the depth informationdetecting section 235 detects the subject distribution in the depthdirection. The process of step S19 may be skipped. In this case, thesubject designation of step S13 remains unaltered, and therefore theprocess of step S20 can be said to be a process that realizes thesubject tracking of the target subjects according to the AF and AEprocesses of step S17.

At step S21, if it is determined that recording stop instructions havenot been received from the image capturer, the control section 201returns to step S16 and processes the next frame. If it is determinedthat recording stop instructions have been received, the process movesto step S22.

At step S22, the moving image generating section 234 connects thecontinuously generated pieces of color image data for the left sideviewpoint and color image data for the right side viewpoint, andperforms a formatting process according to a 3D-compatible moving imageformat such as Blu-ray(registered trademark) 3D to generate the movingimage file. The control section 201 then records the generated movingimage file to the memory card 220 via the memory card IF 207, and thisprocess flow is finished. The recording to the memory card 220 may beperformed sequentially in synchronization with the generation of thecolor image data for the left side viewpoint and color image data forthe right side viewpoint, and an end-of-file process may be performed insynchronization with the recording stop instructions. Furthermore, thecontrol section 201 is not limited to recording on the memory card 220,and may be configured to output data to an external device via a LAN,for example.

The moving image generating section 234 may cause the calculatingsection 233 to perform image processing with a value of 1 for the stereoadjustment parameter C in Expressions 1 and 2 described above, togenerate high definition color image data for the left side viewpointand color image data for the right side viewpoint.

Second Embodiment Example

As a second embodiment example, a process flow for moving imagecapturing that involves changing the stereo adjustment parameter used inthe image processing is described. FIG. 28 shows the process flow duringthe moving image capturing according to the second embodiment example.Processes related to the processes shown in the process flow of FIG. 27are given the same step numbers, such that different processes andadditional processes are described while redundant descriptions areomitted.

In this process flow, step S16 from the flow of FIG. 27 is omitted. Whenthe recording ON instructions are received at step S15, the controlsection 201 performs the AF and AE at step S17 without changing theimaging conditions for parallax amount adjustment. At step S18, thecaptured image data is acquired. The parallax amount between theparallax images in the captured image data acquired here is not withinthe set parallax amount range, according to the imaging conditions andthe subject distribution.

At step S31, the determining section 236 acquires the adjustmentinformation described using FIGS. 24-A to 24-C, and at step S32 thedetermining section 236 references the look-up table using the receivedadjustment information as an argument and determines the value of thestereo adjustment parameter C.

At step S33, the calculating section 233 receives the captured imagedata and the value of the stereo adjustment parameter C determined bythe determining section 236, and generates the color image data of theleft side viewpoint (the RLt_(c) plane data, the GLt_(c) plane data, andthe BLt_(c) plane data) and the color image data of the right sideviewpoint (the RRt_(c) plane data, the GRt_(c) plane data, and theBRt_(c) plane data). The details of this process are described below.

FIG. 29 shows the process flow of step S33, up to the point ofgenerating the parallax color image data that is the color image data ofthe left side viewpoint and the color image data of the right sideviewpoint.

At step S101, the calculating section 233 acquires the captured imagedata. Then, at step S102, as described using FIG. 3, the captured imagedata is divided into planes of the non-parallax image data and theparallax image data. At step S103, as described using FIG. 3, thecalculating section 233 performs the interpolation process tointerpolate the empty pixels in each piece of plane data resulting fromthe division.

At step S104, the calculating section 233 initializes each of thevariables. Specifically, first, the color variable Cset is set to 1. Thecolor variable Cset is such that 1=red, 2=green, and 3=blue.Furthermore, the coordinate variables i and j are both set to 1. Yetfurther, the parallax variable S is set to 1. The parallax variable S issuch that 1=left and 2=right.

At step S105, the calculating section 233 extracts the pixel value fromthe target pixel position (i, j) of the Cset plane. For example, whenCset=1 and the target pixel position is (1, 1), the extracted pixelvalue is Rn₁₁. Furthermore, at step S106, the calculating section 233extracts the pixel values from the target pixel position (i, j) of theLt_(Cset) plane data and the Rt_(Cset) plane data. For example, when thetarget pixel position is (1, 1), the extracted pixel values areLt_(Cset11) and Rt_(Cset11).

At step S107, the calculating section calculates the pixel value for thetarget pixel position (i, j) corresponding to the parallax variable S.For example, when Cset=1, S=1, and the target pixel position is (1, 1),RLt_(C11) is calculated. As a specific example, the calculation may beperformed according to Expression 1 shown above. Here, the stereoadjustment parameter C is the value determined at step S32.

At step S108, the calculating section 233 increments the parallaxvariable S. Then, at step S109, it is determined whether the parallaxvariable S has exceeded 2. If the parallax variable S has not exceeded2, the process returns to step S107. If the parallax variable S hasexceeded 2, the process moves to step S110.

At step S110, the calculating section 233 sets the parallax variable Sto 1 and increments the coordinate variable i. Then, at step S111, it isdetermined whether the coordinate variable i has exceeded i₀. If thecoordinate variable i has not exceeded i₀, the process returns to stepS105. If the coordinate variable i has exceeded i₀, the process moves tostep S112.

At step S112, the calculating section 233 sets the coordinate variable ito 1 and increments the coordinate variable j. Then, at step S113, it isdetermined whether the coordinate variable j has exceeded j₀. If thecoordinate variable j has not exceeded j₀, the process returns to stepS105. If the coordinate variable j has exceeded j₀, the process moves tostep S114.

Upon reaching step S114, all of the pixel values on the right and leftfor this Cset have been handled, and therefore the calculating section233 arranges these pixel values to generate the plane image data. Forexample, when Cset=1, the RLt_(c) plane data and RRt_(c) plane data aregenerated.

The process moves to step S115, and the calculating section 233 sets thecoordinate variable j to 1 and increments the color variable Cset. Then,at step S116, it is determined whether the color variable Cset exceeds3. If the color variable Cset does not exceed 3, the process returns tostep S105. If the color variable Cset exceeds 3, then all of the colorimage data of the left side viewpoint (the RLt_(c) plane data, theGLt_(c) plane data, and the BLt_(c) plane data) and the color image dataof the right side viewpoint (the RRt_(c) plane data, the GRt_(c) planedata, and the BRt_(c) plane data) has been collected, and the flowreturns to FIG. 28.

The following describes desirable shapes for the apertures of theaperture mask described using FIG. 2. FIG. 30 is used to describedesirable aperture shapes.

The aperture section 105 of the parallax Lt pixel and the aperturesection 106 of the parallax Rt pixel are preferably shifted in oppositedirections from each other and include the centers of the correspondingpixels. Specifically, the aperture section 105 and the aperture section106 preferably each contact a virtual center line 322 that passesthrough the center of the pixel or straddles this center line 322.

In particular, as shown in the drawing, the shape of the aperturesection 105 and the shape of the aperture section 106 are preferablyrespectively identical to a shape obtained by dividing the shape of theaperture section 104 of the non-parallax pixel along the center line322. In other words, the shape of the aperture section 104 is preferablyequivalent to a shape obtained by placing the shape of the aperturesection 105 and the shape of the aperture section 106 adjacent to eachother.

In the above description, the arithmetic expressions used by thecalculating section 233 are Expressions 1 and 2, which use a weightedarithmetic mean, but the arithmetic expressions are not limited to thisand various other expressions can be used. For example, when using aweighted geometric mean, Expressions 3 and 4 shown above and expressedin the same manner as Expressions 1 and 2 can be used. In this case, themaintained blur amount is not the blur amount caused found in the outputof the non-parallax pixels, but is instead the blur amount found in theoutput of the parallax pixels.

Furthermore, as other examples of arithmetic expressions, Expressions 5and 6 shown above and expressed in the same manner as Expressions 1 and2 can be used. In this case, the terms of the cube root do not changewhen calculating each of GLT_(cmn), GRt_(cmn), BLt_(cmn), and BRt_(cmn).

Yet further, Expressions 7 and 8 shown above may be adopted. In thiscase as well, the terms of the cube root do not change when calculatingeach of GLt_(cmn), GRt_(cmn), BLt_(cmn), and BRt_(cmn).

The following describes the connection with the display apparatus. FIG.31 is used to describe the connection between the digital camera 10 anda TV monitor 80. The TV monitor 80 is formed by the display section 40made from liquid crystal, a memory card IF 81 that receives the memorycard 220 that is removed from the digital camera 10, and a remotecontrol 82 that is manipulated by a hand of the user, for example. TheTV monitor 80 is adapted to display a 3D image. The display format ofthe 3D image is not particularly limited. For example, the right eyeimage and the left eye image may be displayed at separate times, or aninterlaced format may be used in which the right and left eye images areformed as thin vertical or horizontal stripes. As another example, theright and left eye images may be arranged in a side by side manner indifferent regions of the screen.

The TV monitor 80 decodes the formatted moving image file that includesthe color image data of the left side viewpoint and the color image dataof the right side viewpoint, and displays the 3D image in the displaysection 40. In this case, the TV monitor 80 fulfills the functions of ageneral display apparatus that displays a standardized moving imagefile.

However, the TV monitor 80 can also function as an image processingdevice that realizes at least a portion of the functions of the controlsection 201 described using FIG. 1 and at least a portion of thefunctions of the image processing device 205. Specifically, thedetermining section 236 described in FIG. 1 and the image processingdevice including the calculating section 233 and the moving imagegenerating section 234 may be incorporated in the TV monitor 80. Withthis configuration, it is possible to realize functional roles otherthan the functional roles realized by the combination of the digitalcamera 10 of the second embodiment example and the TV monitor 80.

Specifically, the digital camera 10 associates the depth informationdetected by the depth information detecting section 235 with thegenerated captured image data, without performing the image processingusing the stereo adjustment parameter. The TV monitor 80 serves as animage processing device to reference the associated depth information,determine the value of the stereo adjustment parameter C, and performimage processing using the stereo adjustment parameter on the acquiredimage data. The TV monitor 80 displays in the display section 40 the 3Dimage whose parallax amount has been adjusted in this manner.

The modification described above may be configured such that duringplayback on the TV monitor 80 the viewer can input a portion of theadjustment conditions. For example, the viewer can input the parallaxamount range by manipulating the remote control 82. The TV monitor 80acquires the input parallax amount range as an adjustment condition, andthe determining section 236 determines the value of the stereoadjustment parameter C according to this parallax amount range. Withthis configuration, the TV monitor 80 can display a 3D image accordingto the preferences of each viewer.

In the present embodiment described above, the TV monitor 80 isdescribed as one example of an image processing device, but the imageprocessing device can take on a variety of forms. For example, the imageprocessing device may be a device that includes a display section or isconnected to a display section, such as a PC, mobile telephone, or gamedevice.

In the present embodiment described above it is assumed that a movingimage is being captured, but the configuration for outputting theparallax image data in which the parallax amount is adjusted based onthe detected depth information could obviously also be applied to stillimage capturing. A still image captured in this way does not cause anextreme parallax between the left and right images, and therefore doesnot cause a sense of unnaturalness in the viewer.

In the present embodiment described above, the target subjects arereceived in user instructions, but the control section 201 mayautomatically select target subjects. For example, the control section201 can set the target subjects by using a human recognition process andlimiting the target subjects to images of people included in the scene.

Each process flow of the present embodiment described above is performedaccording to a control program for controlling the control section. Thecontrol program is recorded in an internal nonvolatile memory and isexpanded as needed in a work memory to perform each process. As anotherexample, a control program recorded on a server is transmitted to eachapparatus via a network and expanded in a work memory to perform eachprocess. As yet another example, a control program recorded on a serveris executed on the server and each apparatus performs the processesaccording to control signals transmitted thereto via a network.

While the embodiments of the present invention have been described, thetechnical scope of the invention is not limited to the above describedembodiments. It is apparent to persons skilled in the art that variousalterations and improvements can be added to the above-describedembodiments. It is also apparent from the scope of the claims that theembodiments added with such alterations or improvements can be includedin the technical scope of the invention.

The operations, procedures, steps, and stages of each process performedby an apparatus, system, program, and method shown in the claims,embodiments, or diagrams can be performed in any order as long as theorder is not indicated by “prior to,” “before,” or the like and as longas the output from a previous process is not used in a later process.Even if the process flow is described using phrases such as “first” or“next” in the claims, embodiments, or diagrams, it does not necessarilymean that the process must be performed in this order.

LIST OF REFERENCE NUMERALS

-   -   10: digital camera, 20: imaging lens, 21: optical axis, 22:        diaphragm, 50: eyes, 51: right eye, 52: left eye, 40: display        section, 61, 62, 71, 72: subject, 80: TV monitor, 81: memory        card IF, 82: remote control, 100: image capturing element, 104,        105, 106: aperture section, 110: basic grid, 201: control        section, 202: A/D conversion circuit, 203: memory, 204: driving        section, 205: image processing device, 207: memory card IF, 208:        manipulating section, 209: display section, 210: LCD drive        circuit, 231: adjustment condition acquiring section, 232:        adjustment value determining section, 233: calculating section,        234: moving image generating section, 235: depth information        detecting section, 236: determining section, 300: tree, 301:        girl, 302: boy, 303: woman, 310: AF frame, 320: title, 322:        center line, 1610, 1611: contrast curve, 1620, parallax amount        curve, 1630: adjusted parallax amount curve, 1804, 1805, 1807,        1808: distribution curve, 1806, 1809: combined distribution        curve, 1901: Lt distribution curve, 1902: Rt distribution curve,        1903: 2D distribution curve, 1905: adjusted Lt distribution        curve, 1906: adjusted Rt distribution curve, 2081: dial button,        2082: mode button, 2083: touch panel

What is claimed is:
 1. An imaging apparatus comprising: a processorconfigured to perform operations including generating, by capturing asingle scene, captured image data including reference image data, firstparallax image data having a first parallax in one direction relative toa subject image of the reference image data, and second parallax imagedata having a second parallax in another direction that is opposite theone direction, acquiring an adjustment condition relating to parallaxamount adjustment, and processing the reference image data, the firstparallax image data, and the second parallax image data, based on theadjustment condition, to generate third parallax image data having athird parallax that is in the one direction and different from the firstparallax and fourth parallax image data having a fourth parallax that isin the other direction and different from the second parallax, theprocessing including determining an adjustment value based on theadjustment condition, extracting a reference pixel value at a targetpixel position in the reference image data, a first parallax pixel valueat the target pixel position in the first parallax image data, and asecond parallax pixel value at the target pixel position in the secondparallax image data, and calculating a third parallax pixel valueforming the third parallax image data and a fourth parallax pixel valueforming the fourth parallax image data by an averaging operation usingthe reference pixel value, the first parallax pixel value, the secondparallax pixel value and the adjustment value.
 2. The imaging apparatusaccording to claim 1, wherein the processing further includes generatinga moving image file based on the third parallax image data and thefourth parallax image data by processing the captured image data, whichis generated continuously.
 3. The imaging apparatus according to claim1, wherein the processor is further configured to: output the adjustmentcondition in association with the captured image data.
 4. The imagingapparatus according to claim 3, wherein the outputting includesassociating the adjustment condition with a moving image file createdfrom the captured image data, which is generated continuously.
 5. Theimaging apparatus according to claim 2, wherein the acquiring includesdynamically acquiring the adjustment condition in correspondence withthe captured image data, which is generated continuously.
 6. The imagingapparatus according to claim 1, wherein the adjustment conditionincludes a setting condition of an optical system that focuses thesubject image.
 7. The imaging apparatus according to claim 6, whereinthe setting condition includes at least one of diaphragm constriction,focal distance, and focus lens position.
 8. The imaging apparatusaccording to claim 1, wherein the adjustment condition includes asubject state in the single scene.
 9. The imaging apparatus according toclaim 8, wherein the subject state includes a subject distribution in adepth direction.
 10. The imaging apparatus according to claim 1, whereinthe adjustment condition includes user instructions designated by auser.
 11. The imaging apparatus according to claim 10, wherein theadjustment condition includes a range setting for a parallax amountdesignated by the user instructions.
 12. An imaging apparatuscomprising: a processor configured to perform operations including:generating, by capturing a single scene, captured image data includingreference image data, first parallax image data having a first parallaxin one direction relative to a subject image of the reference imagedata, and second parallax image data having a second parallax in anotherdirection that is opposite the one direction, acquiring an adjustmentcondition relating to parallax amount adjustment, and processing thereference image data, the first parallax image data, and the secondparallax image data, based on the adjustment condition, to generatethird parallax image data having a third parallax that is in the onedirection and different from the first parallax and fourth parallaximage data having a fourth parallax that is in the other direction anddifferent from the second parallax; wherein the adjustment conditionincludes a cumulative parallax amount obtained by integrating a parallaxamount of each pixel in an entire frame.
 13. The imaging apparatusaccording to claim 1, wherein the processing includes calculating thethird parallax pixel value forming the third parallax image data and thefourth parallax pixel value forming the fourth parallax image data byusing the arithmetic expressions of:P ₃=2P ₀×(C·P ₁+(1−C)P ₂)/(P ₁ +P ₂) andP ₄=2P ₀×((1−C)P ₁ +C·P ₂)/(P ₁ +P ₂), where P₀ is the reference pixelvalue, P₁ is the first parallax pixel value, P₂ is the second parallaxpixel value, P₃ is the third parallax pixel value, P₄ is the fourthparallax pixel value, C is the adjustment value, and C is a real numbersuch that 0.5<C<1.
 14. The imaging apparatus according to claim 1,wherein the processing includes calculating the third parallax pixelvalue forming the third parallax image data and the fourth parallaxpixel value forming the fourth parallax image data by using thearithmetic expressions of:P ₃ =P ₀×(P ₁ /P ₂)^((C-0.5)) andP ₄ =P ₀×(P ₂ /P ₁)^((C-0.5)), where P₀ is the reference pixel value, P₁is the first parallax pixel value, P₂ is the second parallax pixelvalue, P₃ is the third parallax pixel value, P₄ is the fourth parallaxpixel value, C is the adjustment value, and C is a real number such that0.5<C<1.
 15. An image processing device comprising: a processorconfigured to perform operations including obtaining captured image dataincluding reference image data, first parallax image data having a firstparallax in one direction relative to a subject image of the referenceimage data, and second parallax image data having a second parallax inanother direction that is opposite the one direction, and an adjustmentcondition relating to a parallax amount adjustment associated with thecaptured image data, and processing the reference image data, the firstparallax image data, and the second parallax image data, based on theadjustment condition, to generate third parallax image data having athird parallax that is in the one direction and different from the firstparallax and fourth parallax image data having a fourth parallax that isin the other direction and different from the second parallax, theprocessing including determining an adjustment value based on theadjustment condition, extracting a reference pixel value at a targetpixel position in the reference image data, a first parallax pixel valueat the target pixel position in the first parallax image data, and asecond parallax pixel value at the target pixel position in the secondparallax image data, and calculating a third parallax pixel valueforming the third parallax image data and a fourth parallax pixel valueforming the fourth parallax image data by an averaging operation usingthe reference pixel value, the first parallax pixel value, the secondparallax pixel value and the adjustment value.
 16. The image processingdevice according to claim 15, wherein the processing further includesgenerating a moving image file based on the third parallax image dataand the fourth parallax image data by processing the captured imagedata, which is continuously obtained.
 17. The image processing deviceaccording to claim 15, wherein the processing further includescalculating the third parallax pixel value forming the third parallaximage data and the fourth parallax pixel value forming the fourthparallax image data by using the arithmetic expressions of:P ₃=2P ₀×(C·P ₁+(1−C)P ₂)/(P ₁ +P ₂) andP ₄=2P ₀×((1−C)P ₁ +C·P ₂)/(P ₁ +P ₂), where P₀ is the reference pixelvalue, P₁ is the first parallax pixel value, P₂ is the second parallaxpixel value, P₃ is the third parallax pixel value, P₄ is the fourthparallax pixel value, C is the adjustment value, and C is a real numbersuch that 0.5<C<1.
 18. The image processing device according to claim15, wherein the processing further includes calculating the thirdparallax pixel value forming the third parallax image data and thefourth parallax pixel value forming the fourth parallax image data byusing the arithmetic expressions of:P ₃ =P ₀×(P ₁ /P ₂)^((C-0.5)) andP ₄ =P ₀×(P ₂ /P ₁)^((C-0.5)), where P₀ is the reference pixel value, P₁is the first parallax pixel value, P₂ is the second parallax pixelvalue, P₃ is the third parallax pixel value, P₄ is the fourth parallaxpixel value, C is the adjustment value, and C is a real number such that0.5<C<1.
 19. A non-transitory computer readable medium for storing acontrol program for an imaging apparatus that causes a computer to:generate, by capturing a single scene, captured image data includingreference image data, first parallax image data having a first parallaxin one direction relative to a subject image of the reference imagedata, and second parallax image data having a second parallax in anotherdirection that is opposite the one direction; acquire an adjustmentcondition relating to parallax amount adjustment; and process thereference image data, the first parallax image data, and the secondparallax image data, based on the adjustment condition, to generatethird parallax image data having a third parallax that is in the onedirection and different from the first parallax and fourth parallaximage data having a fourth parallax that is in the other direction anddifferent from the second parallax, the processing including:determining an adjustment value based on the adjustment condition,extracting a reference pixel value at a target pixel position in thereference image data, a first parallax pixel value at the target pixelposition in the first parallax image data, and a second parallax pixelvalue at the target pixel position in the second parallax image data,and calculating a third parallax pixel value forming the third parallaximage data and a fourth parallax pixel value forming the fourth parallaximage data by an averaging operation using the reference pixel value,the first parallax pixel value, the second parallax pixel value and theadjustment value.
 20. The non-transitory computer readable mediumaccording to claim 19, wherein the control program further causes acomputer to: output the adjustment condition in association with thecaptured image data.
 21. A non-transitory computer readable medium forstoring a control program for an image processing device that causes acomputer to: obtain captured image data including reference image data,first parallax image data having a first parallax in one directionrelative to a subject image of the reference image data, and secondparallax image data having a second parallax in another direction thatis opposite the one direction, and an adjustment condition relating to aparallax amount adjustment associated with the captured image data; andprocess the reference image data, the first parallax image data, and thesecond parallax image data, based on the adjustment condition, togenerate third parallax image data having a third parallax that is inthe one direction and different from the first parallax and fourthparallax image data having a fourth parallax that is in the otherdirection and different from the second parallax, the processingincluding: determining an adjustment value based on the adjustmentcondition, extracting a reference pixel value at a target pixel positionin the reference image data, a first parallax pixel value at the targetpixel position in the first parallax image data, and a second parallaxpixel value at the target pixel position in the second parallax imagedata, and calculating a third parallax pixel value forming the thirdparallax image data and a fourth parallax pixel value forming the fourthparallax image data by an averaging operation using the reference pixelvalue, the first parallax pixel value, the second parallax pixel valueand the adjustment value.
 22. An imaging apparatus comprising: aprocessor configured to perform operations including detecting a subjectdistribution in a depth direction for a scene, determining a changecondition relating to a parallax amount, based on the subjectdistribution, and performing imaging control to generate captured imagedata including first parallax image data and second parallax image datahaving a parallax therebetween, based on the change condition, whereinthe determining includes determining an adjustment parameter value foradjusting a parallax amount between images used when the first parallaximage data and the second parallax image data are generated, as thechange condition, the performing includes acquiring reference imagedata, third parallax image data having a third parallax in one directionrelative to a subject image of the reference image data, and fourthparallax image data having a fourth parallax in another directionopposite the one direction, and a first parallax pixel value iscalculated to generate the first parallax image data and a secondparallax pixel value is calculated to generate the second parallax imagedata by extracting a reference pixel value at a target pixel position inthe reference image data, a third parallax pixel value at the targetpixel position in the third parallax image data, and a fourth parallaxpixel value at the target pixel position in the fourth parallax imagedata, and using an averaging operation using the reference pixel value,the first parallax pixel value, the second parallax pixel value and theadjustment parameter value.
 23. The imaging apparatus according to claim22, comprising: an image capturing element including a first parallaxpixel group that receives, from among incident light passing through anoptical system, first partial light that is polarized in a firstdirection relative to an optical axis of the optical system and a secondparallax pixel group that receives, from among the incident lightpassing through the optical system, second partial light that ispolarized in a second direction different from the first direction. 24.The imaging apparatus according to claim 22, wherein the detectingincludes detecting the subject distribution based on phase differenceinformation for each subject.
 25. The imaging apparatus according toclaim 22, wherein the processor is further configured to: receive adesignation of a subject by a user, wherein the detecting includessetting the designated subject to be a detection target in the subjectdistribution.
 26. The imaging apparatus according to claim 22, whereinthe determining includes determining a diaphragm value as the changecondition.
 27. The imaging apparatus according to claim 22, wherein thedetermining includes determining a correction amount for a focus lensposition as the change condition.
 28. The imaging apparatus according toclaim 22, wherein the first parallax pixel value is calculated togenerate the first parallax image data and the second parallax pixelvalue is calculated to generate the second parallax image data by usingthe arithmetic expressions of:P ₁=2P ₀×(C·P ₃+(1−C)P ₄)/(P ₃ +P ₄) andP ₂=2P ₀×((1−C)P ₃ +C·P ₄)/(P ₃ +P ₄), where P₀ is the reference pixelvalue, P₁ is the first parallax pixel value, P₂ is the second parallaxpixel value, P₃ is the third parallax pixel value, P₄ is the fourthparallax pixel value, C is the adjustment parameter value, and C is areal number such that 0.5<C<1.
 29. The imaging apparatus according toclaim 22, wherein the first parallax pixel value is calculated togenerate the first parallax image data and the second parallax pixelvalue is calculated to generate the second parallax image data by usingthe arithmetic expressions of:P ₁ =P ₀×(P ₃ /P ₄)^((C-0.5)) andP ₂ =P ₀×(P ₄ /P ₃)^((C-0.5)), where P₀ is the reference pixel value, P₁is the first parallax pixel value, P₂ is the second parallax pixelvalue, P₃ is the third parallax pixel value, P₄ is the fourth parallaxpixel value, C is the adjustment parameter value, and C is a real numbersuch that 0.5<C<1.
 30. A non-transitory computer readable medium forstoring a control program for an imaging apparatus causing a computerto: detect a subject distribution in a depth direction for a scene;determine a change condition relating to a parallax amount, based on thesubject distribution; and generate captured image data including firstparallax image data and second parallax image data having a parallaxtherebetween, based on the change condition, wherein the determining thechange condition includes determining an adjustment parameter value foradjusting a parallax amount between images used when the first parallaximage data and the second parallax image data are generated, as thechange condition, the generating captured image data includes acquiringreference image data, third parallax image data having a third parallaxin one direction relative to a subject image of the reference imagedata, and fourth parallax image data having a fourth parallax in anotherdirection opposite the one direction, the generating captured image datafurther includes calculating a first parallax pixel value to generatethe first parallax image data and a second parallax pixel value togenerate the second parallax image data by extracting a reference pixelvalue at a target pixel position in the reference image data, a thirdparallax pixel value at the target pixel position in the third parallaximage data, and a fourth parallax pixel value at the target pixelposition in the fourth parallax image data, and using an averagingoperation using the reference pixel value, the first parallax pixelvalue, the second parallax pixel value and the adjustment parametervalue.
 31. The imaging apparatus according to claim 1, wherein theadjustment value is a real number C such that 0.5<C<1.